Ideas & Insights / All Grantee Impact Stories / Ideas & Insights Grantee Impact Story

Tracking State Reporting of Covid-19 Vaccine Breakthrough Infections

Michal Mart — Data Quality Lead, Pandemic Tracking Collective
Rebecca Glassman, MPH — Senior Public Health Researcher and Data Quality Lead, Pandemic Tracking Collective
Jonathan Gilmour — Data Scientist, Pandemic Tracking Collective

Today, we are releasing a reporting scorecard that evaluates U.S. state data reporting for Covid-19 vaccine breakthrough infections (BI) against best practices we have identified. We assign states letter grades based on their alignment with these best practices for reporting BI data and will update the grades monthly to monitor progress. The scorecard with states’ grades, as well as breakdowns of the criteria they meet, can be found here.

We are launching this scorecard due to the lack of high-quality breakthrough data in the United States. While the CDC shares some helpful information on BI rates, the information is challenging to aggregate on a nationwide scale; as a result, it is updated infrequently and cannot be broken down by state. State-provided BI data is faster and more granular than federal data, but like with other state Covid-19 metrics, largely incomplete and unstandardized. The ability to see local, fast-updating data on vaccine breakthrough infections is critical to an effective pandemic response. We hope the scorecard will encourage states to share more and higher-quality vaccine breakthrough data in support of that mission.

What we’re grading

Our scorecard evaluates public BI data reporting for all 50 states, as well as the District of Columbia and Puerto Rico. We assign states whole letter grades A through F; if states meet every criterion we have identified, they receive an A+. This letter grade solely reflects states’ alignment with BI reporting best practices and does not evaluate how well a state is responding to Covid-19 overall.

We break down best practices for reporting into two categories: data reporting, which addresses what states are sharing, and data presentation, addressing how they are sharing it. These criteria will be reviewed periodically to ensure they reflect the most pressing priorities for data on vaccine breakthrough infections—for example, we may begin tracking reporting of infections after vaccine “booster” doses. Here’s a summary of the criteria we are including in each category and why they are important. You can find details of how they are evaluated on the scorecard page.

Data Reporting

  • Providing cumulative vaccine breakthrough cases, breakthrough hospitalizations, breakthrough deaths, and vaccinations: When a state can produce these cumulative counts, it indicates the data reporting systems it uses to produce vaccination status breakdowns are in good order.
  • Providing cumulative percentages of Covid-19 cases, hospitalizations, and deaths among vaccinated and unvaccinated individuals: These percentages are important for public communication about the absolute and relative risk of Covid-19 infection, hospitalization, or death in vaccinated and unvaccinated populations.
  • Providing time series of cases, hospitalizations, and deaths in vaccinated and unvaccinated people: Case, hospitalization, and deaths data by vaccination status over time are perhaps the most important breakthrough data for both public and policymakers alike—they allow for tracking absolute and relative changes in the incidence and severity of breakthrough Covid-19 infections. As with cumulative data, states can report time series data either as raw numbers of outcomes in vaccinated/unvaccinated population and the number of people in that population, or as rates dividing those two numbers. But rates can be calculated in many different ways, so without access to underlying data, they can be impossible to replicate or interpret. For that reason, for a state to receive the highest possible score for a time series, states had to report both a rate and all underlying data that went into that rate (i.e. number vaccinated over time and number of breakthrough cases over time).

Data Presentation

  • Providing definitions for vaccine breakthrough cases, breakthrough hospitalizations, breakthrough deaths, vaccinations, and rates: Not all states define their Covid-19 metrics the same way, so it is important for states to provide definitions of all metrics.
  • Making downloadable data available: Providing downloadable data, for example, as comma-separated values (CSV), increases transparency and enables researchers to use it in their own analysis.
  • Regularly reporting data: Because timely data is important for understanding trends, states that do not share data at least monthly automatically receive an F.

What we found

Figure 1:  Currently, nine states receive an F grade because they do not share data regularly. Of the remaining states, three (California, Colorado, and Utah) meet the criteria for an A.

Our analysis of state reporting found that most states are now reporting breakthrough data—with 43 out of 52 jurisdictions now regularly sharing data, up from just over half that in the summer. Almost all of those states report important cumulative counts of cases, hospitalizations, and deaths; 34 of them report at least one metric over time.

Figure 2: Including states with both regular and ad hoc public reports, as of December 16, 2021, 40 states have reported cumulative breakthrough cases, 37 states have reported cumulative breakthrough hospitalizations, and 42 states have reported cumulative breakthrough deaths.

The number of states that are reporting data is impressive given the uphill battle that state health departments have faced in collecting vaccine breakthrough data. Over the past decades, state health departments have been chronically underfunded, resulting in out-of-date, badly-maintained data systems for infectious disease surveillance. When we contacted states to confirm our assessments, some shared that they do not have resources to compile regular updates. To regularly capture, link, and publish these critical data, states need federal support, like that provided through the CDC’s Data Modernization Initiative or upcoming data infrastructure projects under the American Rescue Plan.

Even given infrastructure obstacles, an easy way many states could improve their grade is by making downloads available of all data underlying their time series—data they already use to calculate rates. While 34 states report at least one metric (i.e. incidence, hospitalization or mortality rates) by vaccination status over time, only three states make underlying data available for all reported rates, and only six states make any data downloadable. Currently, only three states meet the standard for an A [See Fig. 1]; if states made underlying data downloads available for all rates they are already reporting, 17 additional states would receive this grade.

Why it matters

Throughout our experience collecting Covid-19 data, we have repeatedly seen that providing consistent, interpretable, high quality data garners public trust. BI data shows a clear trend that Covid-19 hospitalization and deaths are largely concentrated in unvaccinated individuals. Sharing those data plays an important role in overcoming vaccine hesitancy.

Vaccine breakthrough data has other urgent policy implications: It provides crucial early warning signals for possible changes in transmission, vaccine-conferred immunity, and disease dynamics. To understand the risk that vaccinated individuals face from variants like Delta and Omicron—and to assess impact on vaccine effectiveness of any new variants that may emerge—we need access to timely and local vaccine breakthrough data that states can provide. 

Working with limited resources, over the past two years, state health officials have accomplished remarkable feats to generate and share critical Covid-19 data with the public. We are optimistic that they will continue this exemplary public service by improving Covid-19 breakthrough data so that it is more transparent, standardized, and actionable.