How Flu Vaccine Effectiveness and Efficacy are Measured

How do we measure how well flu vaccines work?

Two general types of studies are used to determine how well flu vaccines work: randomized controlled trials and observational studies. These study designs are described below.

Randomized controlled trials (RCTs)

The first type of study is called a randomized controlled trial (RCT). In an RCT, volunteers are assigned randomly to receive a flu vaccine or a placebo (e.g., a shot of saline). RCTs measure vaccine efficacy. Vaccine efficacy refers to the percent reduction in the frequency of flu illness among vaccinated people compared to people who are not vaccinated. Vaccine efficacy is measured in RCTs, usually under optimal conditions where vaccine storage and delivery are monitored, and participants are usually in good health or selected for a specific health status. Vaccine efficacy is measured by comparing how often people in the vaccinated and the unvaccinated (placebo) groups get flu. The RCT study design minimizes bias that could lead to invalid study results. Bias is an unintended systematic error in the way researchers select study participants, measure outcomes, or analyze data that can lead to inaccurate results. In an RCT, vaccine assignments are usually double-blinded, which means neither the study volunteers nor the researchers know if a given person has received vaccine or placebo. National regulatory authorities, such as the Food and Drug Administration (FDA) in the United States, require RCTs to be conducted and to demonstrate the protective benefits of a new vaccine before the vaccine is licensed for routine use. However, some vaccines are licensed based on RCTs that use antibody response to the vaccine as measured in the laboratory, rather than decreases in flu disease among people who were vaccinated.

Observational Studies

The second type of study is an observational study. There are several types of observational studies, including cohort and case-control studies. Observational studies measure vaccine effectiveness by assessing how well flu vaccines work among different groups of people, in different settings, and in different real-world conditions that exist outside of randomized controlled trials (i.e., clinical trials). Vaccine effectiveness is measured by comparing how often people in the vaccinated and unvaccinated (placebo) groups get flu. Vaccine effectiveness is the percent reduction in the frequency of flu illness among vaccinated people compared to people not vaccinated, usually with adjustment for factors  that are related to both flu illness and vaccination (e.g., the presence of chronic medical conditions). (See below for more details.)

Top of Page

How do vaccine effectiveness studies differ from vaccine efficacy studies?

Vaccine efficacy refers to vaccine protection measured under optimal conditions where vaccine storage and delivery are monitored and participants are usually healthy. Vaccine efficacy is determined in RCTs, usually clinical trials. Vaccine effectiveness measures how well a vaccine works in real-world conditions. Differences in real-world conditions compared to the tightly controlled conditions in clinical trials can influence how well a vaccine works. Vaccine effectiveness studies include people with underlying medical conditions who have been administered vaccines by different health care providers. Vaccine effectiveness studies can be used to determine if higher risk groups of people (often excluded from clinical trials/RCTs) respond differently to the vaccine. Vaccine effectiveness studies can also determine if different flu viruses that are circulating and evolving in real-world conditions affect vaccine performance. Lastly, vaccine dosing schedules or vaccine storage and handling requirements may not be followed as closely in the real world as in clinical trials; therefore, vaccine effectiveness studies can be conducted to account for how these factors affect vaccine performance. Results from vaccine effectiveness studies are subject to biases that do not occur in vaccine efficacy studies, like selection bias and confounding, which is why licensure of vaccines depends upon data collected in RCTs.

Once a flu vaccine has been licensed by FDA, recommendations are typically made by CDC’s Advisory Committee for Immunization Practices (ACIP) for its routine use. For example, ACIP now recommends annual flu vaccination for all U.S. residents aged 6 months and older, with rare exception. Universal vaccine recommendations introduce ethical challenges in performing RCTs which assign people to a placebo group, which could place them at risk for serious complications from flu. Also, observational studies often are the only option to measure vaccine effectiveness against more severe, less common flu outcomes, such as hospitalization.

Top of Page

What factors can affect the results of flu vaccine efficacy and effectiveness studies?

The measurement of flu vaccine efficacy and effectiveness can be affected by virus and host factors as well as the type of study used. Therefore, vaccine efficacy/effectiveness point estimates have varied among published studies.

Virus factors

The protective benefits of flu vaccination are generally lower during flu seasons where the majority of circulating flu viruses differ from the flu viruses used to make the vaccines. Flu viruses are continuously changing through a natural process known as antigenic drift. (For more information, see How the flu virus can change: Drift and Shift.) However, the extent of antigenic drift and the number of drifted viruses in circulation can vary for each of the four viruses included in the seasonal flu vaccine. So even when circulating flu viruses are mildly or moderately drifted in comparison to the vaccine, it is possible that people may still receive some protective benefit from vaccination; and if other circulating flu viruses are well matched, the vaccine could still provide protective benefits overall.

Host factors

In addition to virus factors, host factors such as age, underlying medical conditions, history of prior flu illness and prior flu vaccinations can affect the benefits received from vaccination.

Study Design Factors

RCTs provide the most reliable results because they are less susceptible to biases, including selection bias and confounding. However, as stated above, RCTs may be difficult to conduct when vaccination is recommended in a population or for more severe outcomes that are less common, given the large numbers of people that would have to be randomized. There are several observational study designs; however, many flu vaccine evaluation programs currently use the test-negative design. In the test-negative design, people who seek care for an acute respiratory illness are enrolled at care settings (such as outpatient clinics, urgent care clinics, emergency departments, or in in-patient settings) and information is collected about the patients’ flu vaccination status. All participants in a test-negative design study are tested for flu using a highly specific and sensitive test for flu virus infection, such as reverse transcription polymerase chain reaction (RT-PCR). The ratio of vaccinated to unvaccinated persons (i.e., the odds of flu vaccination) is compared among patients with and without laboratory-confirmed flu. In this way, a test-negative design study estimates VE by comparing vaccination rates among persons with confirmed flu illness (also called “cases”) versus persons with similar illness who do not have flu (also called “controls”) based on laboratory tests. The test-negative design reduces selection bias due to healthcare seeking behaviors. Other observational study designs have also been used to estimate flu vaccine effectiveness.

Factors Related to Measuring Specific versus Non-Specific Outcomes

For both RCTs and observational studies, the specificity of the outcome measured in the study is important. Non-specific outcomes, such as pneumonia hospitalizations or influenza-like illness (ILI) can be caused by flu virus infections or infections with other viruses and bacteria. Vaccine efficacy/effectiveness estimates for non-specific outcomes are generally lower than estimates made for more specific outcomes, depending on what proportion of the outcome measured is attributable to flu. For example, a study among healthy adults found that the inactivated flu vaccine (i.e., the flu shot) was 86% effective against laboratory-confirmed flu, but only 10% effective against all respiratory illnesses in the same population and season [1]. Laboratory-confirmed flu virus infections, by RT-PCR or viral culture, are generally the most specific outcomes for vaccine efficacy/effectiveness studies.

Serologic assays to detect flu (i.e., which require a four-fold rise in antibody titers against flu viruses detected from paired sera) were often used in flu VE studies conducted prior to the development of RT-PCR tests. An issue with VE studies that use serology tests for flu virus infection is that vaccination elevates antibody levels as does infection, complicating the interpretation of these studies. New flu virus infections could be missed in a vaccinated person since antibodies are already high and a four-fold increase doesn’t develop. Therefore, serologic testing methods can result in estimates that incorrectly inflate VE.

Top of Page

Can you describe biases that should be considered in observational studies measuring vaccine effectiveness?

Results from observational studies are more likely to be affected by various forms of bias (see above for definition) than are results from RCT studies. Therefore, results from observational studies can be more difficult to interpret. Bias can be reduced through careful study designs and analyses of data collected through observational studies. Observational studies of flu vaccine effectiveness are subject to at least three forms of bias: confounding, selection bias, and information bias.

Confounding occurs when the effect of vaccination on the risk of the outcome being measured (e.g., flu-related hospitalizations confirmed by RT-PCR) is distorted by another factor associated both with vaccination (the exposure) and the outcome. In RCTs, factors associated with exposure and outcomes are evenly distributed between vaccinated and unvaccinated groups. This is not always true in observational studies. For example, chronic medical conditions can confound the association between flu vaccination and hospitalization with flu in observational studies. Chronic medical conditions increase the risk of flu-related hospitalization and vaccination often is more common among people with chronic medical conditions. Therefore, the presence of a chronic medical condition in a study participant is a potential confounding factor that should be considered in analysis. This is an example of confounding by indication because those at greatest risk for the outcome being measured (i.e., flu-related hospitalization) are targeted for vaccination, and therefore, they are more likely than those without a chronic medical condition to receive a flu vaccine. Not adjusting for confounders can bias the vaccine effectiveness estimate higher or lower than the true estimate. In the example given, the vaccine effectiveness estimate could be biased lower, or towards lower effectiveness.

Selection bias occurs when people with the outcome being measured by the study (i.e., flu virus infection) differ from people who do not have the outcome. In observational studies of flu vaccine effectiveness, people with and without flu may have different likelihoods of being vaccinated, and this can bias the estimate of vaccine effectiveness. For example, people who visit their health care provider in outpatient settings (e.g., clinics and urgent care) may be more likely to be vaccinated than people who do not go to a provider for care as often. If controls are selected from a different population than the cases (e.g., cases are from a clinic and controls from a community sample) with different health care seeking behaviors, selection bias related to health care seeking (and the likelihood to be vaccinated) may be introduced. The test-negative study design minimizes selection bias related to health care seeking by enrolling patients who seek care for a respiratory illness. This study design is used by many studies globally, including CDC-funded networks that measure vaccine effectiveness.

Information bias occurs if exposures or outcome information are based on different sources of information for people with and without the disease of interest. For example, if researchers obtain information on vaccination for children with flu from immunization records but ask parents of children without flu if the child was vaccinated, this difference in data collection procedures could bias the results of the study.

Top of Page

How well do flu vaccines work during seasons in which the flu vaccine is not well matched to circulating flu viruses?

As described above, when the virus components of the flu vaccine are not well-matched with circulating flu viruses, the benefits of flu vaccination may be reduced. However, the extent of antigenic drift from vaccine viruses and the proportion of circulating drifted viruses varies each season. As a result, even when circulating flu viruses are mildly or moderately drifted in comparison to the vaccine, it is possible that people may receive some protective benefit from flu vaccination. Even if some circulating flu viruses are significantly drifted, it is possible for other flu viruses in circulation to be well-matched to the vaccine. It is not possible to predict how well the vaccine and circulating strains will be matched in advance of the flu season, nor is it possible to predict how this match may affect vaccine effectiveness.

Top of Page

What is the evidence that flu vaccines work?

Adults

Several RCTs have been done in healthy adults aged <65 years [7,8,9,10,11,12]. These studies have reported vaccine efficacy estimates ranging from 16%-75%; vaccine efficacy of 16% was reported during a season with few flu virus infections. An RCT in South Africa among HIV infected adults reported vaccine efficacy of 76% (95 CI 9-96) [13]. A meta-analysis that included data from RCTs of licensed inactivated flu vaccines reported vaccine efficacy of 59% (95% CI 51-67) against flu confirmed by RT-PCR or viral culture [14]. RCTs of cell-based inactivated flu vaccines (IIVs) and recombinant trivalent HA protein vaccines have been performed among healthy adults. In general, efficacy estimates for these types of vaccines are similar to efficacy of other inactivated flu vaccines that are egg-based [15,16,17]. A meta-analysis of data from five seasons (2010-2014) estimated VE of 41% (95% CI, 34-48) for prevention of flu hospitalizations in adults aged 18 years and older and VE of 51% (95% CI, 44-58) among adults aged 18-64 years. These results suggest that vaccination decreased the risk of severe flu by half in this age group and by one-third among older adults [43].

Children

In a four-year RCT of inactivated vaccines among children aged 1–15 years, vaccine efficacy was estimated at 77% against flu A (H3N2) and 91% against flu A (H1N1) virus infection [18].^. An RCT of children aged 6–24 months reported vaccine efficacy of 66% against laboratory-confirmed flu in 1999-2000 but no vaccine efficacy during the second year when there was little flu activity [19]. During 2010-11, the vaccine efficacy of a quadrivalent inactivated vaccine among children aged 3-8 years was 59% (95% CI: 45%-70%) [20]. A cluster-randomized trial conducted in Hutterite communities in Canada found that vaccinating children aged 3 to 15 years with trivalent inactivated flu vaccine before the 2008-09 season reduced RT-PCR confirmed flu in the entire community by 61% (95% CI: 8-83), including a 59% reduction (95% CI: 5-82) in confirmed flu among non-vaccinated community members. This study provides evidence of an “indirect” effect of flu vaccination on prevention of disease transmission [21].

Several RCTs of live attenuated flu vaccines among young children have demonstrated vaccine efficacy against laboratory confirmed flu with estimates ranging from 74%-94% [22,23,24,25]. A study conducted among children aged 12 through 36 months living in Asia during consecutive flu seasons reported efficacy for live attenuated flu vaccine of 64%–70% [26].

Adults 65 years and older

Among older adults, annual flu vaccination is recommended based on the high burden of flu-related disease and demonstrated vaccine efficacy among younger adults. One RCT of adults aged 60 years and older relied on serology for confirmation of flu and reported a vaccine efficacy of 58% (95% confidence interval (CI): 26-77) [2]. However, it is unknown if infections were missed by serology among the study participants that were vaccinated (and if the vaccine efficacy estimate is biased upwards – see previous description of how bias can occur in vaccine efficacy studies that test for flu using serology). A meta-analysis of observational studies that used the test-negative design provided VE estimates for adults aged >60 years against RT-PCR-confirmed flu infection. This meta-analysis reported vaccine effectiveness of 52% (95% CI: 41-61) during seasons when the vaccine and circulating viruses were well-matched [3]. During seasons when the circulating viruses were antigenically drifted (not well-matched), reported VE was 36% (95% CI: 22-48) [3].

An RCT that compared a high-dose, inactivated flu vaccine (containing four times the standard amount of flu antigen) to standard dose vaccine in persons aged 65 years and older during the 2011-12 and 2012-13 flu seasons found that rates of laboratory-confirmed flu were 24% lower (95% CI: 10-37) among people who received high-dose vaccine compared to standard dose flu vaccine, indicating that high-dose vaccine provided better protection against flu than standard dose vaccine in this trial [4].

Several observational studies have reported vaccine effectiveness against RT-PCR confirmed flu-related hospitalization among older adults. A three-year study (2006-07 through 2008-09) in Tennessee that used a test-negative design reported vaccine effectiveness of 61% (95% CI: 18-83) among hospitalized adults >50 years of age [5]. In an analysis of two additional seasons, including 2010-11 and 2011-2012 (excluding 2009-10), VE was 58% (95% CI: 8-81) against RT-PCR confirmed flu-related hospitalizations for people older than 50 for the five seasons combined [6].

Pregnant people

An RCT conducted among pregnant people in South Africa during 2011 and 2012 reported vaccine efficacy against RT-PCR confirmed flu of 50% among HIV-negative persons and 58% among HIV-positive persons vaccinated during the third trimester [27]. The trial showed that vaccination reduced the incidence of laboratory-confirmed flu among infants born to HIV-negative persons by 49%; the study was unable to demonstrate vaccine efficacy among infants of HIV-infected persons. An observational study in the United States during 2010-11 and 2011-12 using a test-negative design reported vaccine effectiveness of 44% (95% CI: 5 to 67) against flu among pregnant people [28].

Top of Page

How well does the live attenuated flu vaccine (LAIV) work compared to inactivated flu vaccine (IIV)?

Children

Three randomized clinical trials comparing live attenuated flu vaccine to trivalent inactivated flu vaccine in young children, 2-8 years of age, suggested that live attenuated flu vaccine had superior efficacy compared to inactivated flu vaccine [36,37,38]. However, several recent observational studies suggest that LAIV did not consistently provide better protection against flu than inactivated vaccine, especially against flu caused by the 2009 H1N1 pandemic virus [39,40,41]. Results from a randomized, school-based study conducted in Canada showed lower rates of confirmed flu among students vaccinated with live-attenuated vaccine compared with students vaccinated with inactivated flu vaccine, and decreased flu transmission among family members of students vaccinated with live-attenuated flu vaccines [42]. ACIP does not express a preference for live-attenuated or inactivated flu vaccines for children or adults.

Adults

Clinical trials conducted during 2004-05, 2005-06, and 2007-08 that compared inactivated flu vaccines and live attenuated flu vaccines to no vaccine among adults suggested that inactivated flu vaccines provided better protection against flu than live attenuated flu vaccines in adults [7,8].

 

Top of Page

How does CDC monitor vaccine effectiveness?

CDC has been working with researchers at universities and hospitals since the 2003-2004 flu season to estimate how well flu vaccine works through observational studies using laboratory-confirmed flu as the outcome. These studies currently use a very accurate and sensitive laboratory test known as real-time RT-PCR (reverse transcription polymerase chain reaction) to confirm medically attended flu virus infections as a specific outcome. CDC’s studies are conducted in sites located across the United States to gather more representative data. To assess how well the vaccine works across different age groups, CDC’s studies of flu vaccine effects have included all people aged 6 months and older recommended for an annual flu vaccination. Similar studies are being done in Australia, Canada and Europe.

Over the past few years, CDC has conducted VE studies using multiple networks, including the U.S. Flu VE Network, the Hospitalized Adult Flu Vaccine Effectiveness Network (HAIVEN), the Flu and Other Viruses in the Acutely Ill (IVY) network, the National Vaccine Surveillance Network (NVSN), and the VISION VE Network. For this upcoming winter, HAIVEN will no longer be used. HAIVEN looked at how well flu vaccines protect against flu-related hospitalization among adults aged 18 and older. HAIVEN ended enrollment on July 31, 2021, but CDC will continue to collect information on adults hospitalized with flu through its other VE networks, including IVY and VISION. IVY consists of 21 large, adult hospitals in 20 U.S. cities and was originally created in 2019 to estimate how well the flu vaccine works at preventing severe flu illness among intensive care unit (ICU) patients. As of April 1, 2021, IVY has expanded to enroll all adults hospitalized with COVID-19. During the 2021-2022 flu season, the network will also enroll patients hospitalized with flu. NVSN collects vaccine effectiveness data on pediatric hospitalizations with laboratory confirmed flu in children 18 years of age and younger. The VISION VE Network collects data on emergency department visits, hospitalizations, and intensive care unit (ICU) admissions. The network was established in 2019 and includes the following eight U.S. sites:

• Baylor Scott and White Health (BSHW; Texas)
• Columbia University Irving Medical Center (CUIMC; New York)
• HealthPartners (HP; Minnesota and Wisconsin)
• Intermountain Healthcare (IH; Utah)
• Kaiser Permanente Northern California (KPNC; California)
• Kaiser Permanente Northwest (KPNW; Oregon and Washington)
• Regenstrief Institute (RG; Indiana)
• University of Colorado (UCO; Colorado).

Top of Page