Deming's 14 Points and Vaccine Distribution (Part 2)

There’s a new literary genre in newspapers and magazines. I call it “vaccine villainy.” The story starts by describing Mary, who thus far has not been able to obtain a dose of one of the COVID-19 vaccines. Mary is a senior citizen who hasn’t seen her grandchildren for months, or has a high-risk health condition, or works in a job with high exposure to people who may be infected. By the end of the third paragraph, the reader’s sympathies are entirely with Mary—she deserves this vaccine and should be able to get it.

The writer, however, knows that every story needs an antagonist and proceeds to describe one. This may be a younger person who received a “plus-one” dose by accompanying a senior citizen, or a volunteer who was given a dose after directing traffic at the vaccination site, or a pharmacist who gave about-to-expire doses to the persons first in line instead of Mary (even though she was priority group 1B2f and they were priority group 1B3d), or a snowbird who is visiting from another state, or a jail inmate.

But none of these people* are the reason that Mary has been unable to get the vaccine. Statistician W. Edwards Deming repeatedly emphasized that deficiencies of quality (here, the ability to get vaccines to everyone) are properties of the system, and can only be addressed through improving the system via leadership. The workers in the shoe-sewing factory described in Part 1 could not compensate for machine breakdowns caused by cheap thread; only a change in the system, switching to better thread, could do that.

It’s the System

In 2020, two of the components of the system of vaccine production and distribution worked magnificently. The scientific development of the vaccine is likely to be recorded as one of the great achievements of 21st-century medicine. The most optimistic timeline projected by vaccine experts interviewed by The New York Times in April 2020 had a vaccine ready for distribution in August 2021; a typical vaccine development takes 5 to 10 years and the previous record for vaccine development was 4 years for the mumps vaccine. The speed was achieved while still adhering to standard vaccine development protocols by building on previous research and taking steps such as funding multiple vaccine teams and beginning manufacturing in parallel with clinical trials.

Less well publicized is a second part of the system that worked magnificently: the clinical trials testing the vaccines. These were the result of decades of statistical research on how to conduct clinical trials (and, in particular, vaccine trials). In June 2020, the Food and Drug Administration released guidelines for vaccine performance, specifying that a vaccine should reduce disease incidence or severity in at least 50% of subjects, with a lower confidence bound of 30%. The clinical trials not only showed that the Pfizer and Moderna vaccines easily exceeded these benchmarks, but they were conducted in such a way that the high efficacy was demonstrated across age, race/ethnicity, and gender categories and for persons with underlying medical conditions.

Moreover, the statisticians designing the trials knew that an important part of the vaccine campaign, should the vaccines prove effective, would be persuading the public to take them. They designed trials that can be understood by anyone, regardless of statistical training. The Pfizer trial, for example, randomly assigned about half of the 43,000 participants to get the vaccine and the other half to get the placebo, where individual participants (and the people administering the shots) did not know which shot they were given. After follow-up, the vaccine group had 9 symptomatic cases and the placebo group had 169 symptomatic cases. Since each group had roughly the same number of participants, the efficacy (reduction in relative risk) can be approximately calculated as 100 x (1 - 9/169), or about 95%.**

It’s a remarkable achievement to develop and test vaccines within a 12-month period. But vaccines only work if they’re injected into people, and the rest of the system in the US—and particularly the vaccine distribution—did not work well in 2020.

Hence Mary’s frustration. But the problem is not that other people have been getting vaccine doses; it is that (a) there is currently not enough supply and (b) the system for signing up is complicated and hinders equal access. If sign-up is primarily by web site, and Mary does not have internet access, then she will not be able to register. For Mary to be able to get a vaccine, the system needs to be changed.

That was precisely Deming’s argument in his Fourteen Points for Management (Deming, 1986). Let’s look at a few of the points that are especially pertinent to vaccine distribution.

Point 1. Constancy of Purpose

Deming’s first point was for an organization to “create constancy of purpose for improving products and services.” This means defining the purpose of the system and managing it so that all components cooperate toward achieving the purpose.

In December 2020, the CDC’s Advisory Committee on Immunization Practices set out four ethical principles for allocating the initial supplies. The first principle, to maximize benefits and minimize harms, called for “Respect and care for people using the best available data to promote public health and minimize death and severe illness.” The other principles specified a vaccine distribution system that would mitigate health inequities, promote justice, and promote transparency.

Beyond that, however, the allocation and distribution was left up to state public health departments. Different states use different algorithms to determine who is prioritized for vaccine. One county or state may be vaccinating persons 65-74 or grocery workers while a neighbor is not. A health condition such as asthma may be considered a high-risk health condition in one state but not another. The confusion alone creates barriers to COVID-19 vaccination. In some states the primary mode to register for the vaccine has been over the internet, which has made it difficult for persons without internet access, or with slow connections, to obtain appointments.***

But I am optimistic that many of these difficulties will be resolved in the next month or two. Many states have improved their initial registration and distribution systems, and are now set up to deliver as many vaccine doses as they can obtain. The COVID plan published by the Biden administration adopts a systems-based approach, and specifically addresses all aspects of the vaccine distribution system: manufacturing vaccines and associated supplies such as medical glass and syringes, distributing them to where they are needed, focusing on high-risk and hard-to-reach populations, increasing the number of vaccinators, launching a public education campaign to reduce vaccine hesitancy, and publicly sharing data about key responses.

The last aspect of the plan addresses a central feature of Deming’s philosophy: to use data to track progress and make decisions. One of the biggest complaints from states has been the uncertainty about how many doses will be delivered each week. This is one of the problems that statistical process control solves. In the television program If Japan Can, Why Can’t We, Deming said: “In statistical control you have a reproducible product hour after hour, day after day. And see how comforting that is to management: they now know what they can produce, they know what their costs are going to be.”

Point 5. Improve Every Process Constantly and Forever

Deming emphasized that achieving quality is a process, not a once-and-done activity. “Quality must be built in at the design stage” (Deming, 1986, p. 49). He asks: “Is there continual improvement of materials, of selection of new employees, of the skills of people at work on the job, and of repeated operations?”

The continual improvement is driven by data, through a cycle of four repeated steps. First: plan and develop ideas for changes that might improve the process. Second: do tests with the proposed changes, collecting data on the results. Third: study the data. Fourth: act on the results. These activities then generate new ideas, and the cycle continues.

Continual improvement is especially important for vaccine production and distribution. It’s not surprising that the initial production procedures, which were based on procedures used to produce the vaccine in the laboratory, were not optimized for efficiency. Through use of quality improvement procedures, however, Pfizer has managed to cut the time required to make a batch of vaccine from 110 days to 60 days.

Point 11. Eliminate Numerical Goals

Deming’s advice to “eliminate numerical goals” is, I think, the most misunderstood of his 14 points. After all, don’t we want to have goals? And don’t we want to be able to measure progress toward those goals, which involves numerical measurements?

Absolutely, we want to have goals and, in fact, Point 1 emphasizes having constancy of purpose for achieving the aim of the system. But Deming was opposed to numerical goals that are “drawn out of the sky.” He wrote: “A numerical goal accomplishes nothing. What counts is by what method. If you can accomplish a goal without a method, then why weren’t you doing it last year?” (Deming, 2013, p. 55). In fact, without a method, the numerical goal incentivizes people to falsify statistics, since there is no way to reach the goal.

There were numerous examples of numerical goals without methods in 2020. On September 16, two months before the first vaccine trial concluded, the former president promised 100 million vaccine doses by the end of 2020. Later, this figure was revised to promise 20 million vaccinations by the end of 2020. What actually was achieved? 5.4 million doses of vaccine were administered as of December 31, 2020, about 5% of the original numerical goal. The goal was laudable, but there was no method for achieving it.

Deming contrasted numerical goals (unaccompanied by methods) with predictions (based on data about the system’s capabilities). Data from a stable production process can be used to make predictions about output. A vaccine production and distribution system that results in at least 1 million vaccine administrations every day, and that is in statistical control, can be expected to result in at least 100 million vaccine administrations in 100 days. This is not a numerical goal “drawn out of the sky”: It’s a prediction based on data.

Deming also warned about relying on a single metric to measure quality, arguing that this can lead to over-focusing on that metric at the expense of other quality goals. Numerous websites track the percentage of each state’s population that has received at least one dose, or two doses of vaccine. Now, if 100% of the population has received two doses, then the goal of the vaccination campaign will have been achieved. But short of that, an emphasis on that one number can interfere with other goals. There is a great deal of evidence that the vaccination rate thus far is lower among persons of color, or in zip codes with low median income. Thus, a wide range of data needs to be collected and monitored to ensure the goals of equitable vaccine distribution.

Point 14. Put Everybody to Work

Deming’s last point is to involve everyone in the system—that means all of us—to accomplish the transformation.

There is an antagonist in this story, but it’s not the people who have gotten the vaccine earlier than Mary, or the reporter, thinks they should have. They are simply following the advice of Dr. Fauci and other public health experts: “When a vaccine becomes available to you, get vaccinated. You will be making a major step in a positive way to protecting the community.”

The antagonist in this story is the virus. We don’t yet know how the story ends, but the protagonists have a powerful weapon: a systems-based approach driven by data.

Copyright (c) 2021 Sharon L. Lohr

Footnotes

*Sometimes the person getting the vaccine really is a villain, as in the case of the paramedic who stole doses that were meant for first responders. But that still is a problem of a system that did not keep close track of where the vaccines were. One additional problem with the vaccine villainy stories is that they are completely anecdotal. The stories make it seem like the majority of people currently being vaccinated are not yet in an eligibility group, but there are no statistics supporting that impression.

**The actual calculations can be more complicated than this, and the calculation of the confidence bounds more complicated still. But the trial is designed in such a way that this ballpark calculation is pretty close to the estimated efficacy, and it is easy to understand. The 43,000 persons included some who had previously been infected with the virus; of the approximately 37,000 persons who had not previously been infected, 8 in the placebo group contracted COVID compared with 162 in the placebo group, which again is an efficacy of about 95%.

Note that efficacy is the estimated reduction in risk relative to being unvaccinated under the conditions in the clinical trial. It does not tell you your probability of getting COVID after being vaccinated (or not vaccinated). To see this, note that a trial where 900 people in the vaccine group and 16900 people in the placebo group had symptomatic COVID would have the same vaccine efficacy as this trial, but obviously the risk of is much higher if 900 people in the vaccine group had symptoms rather than 9 people.

***Pew Research estimated that although 9 out of 10 Americans used the internet in 2019, that percentage dropped to 73% for adults age 65 and over, to 84% for persons with a high school degree but no college, and to 71% for persons with no high school degree. Even fewer had high-speed broadband at home, the type of connection that would be needed to rapidly refresh the screen and grab appointments. 73% overall, but only 59% of those age 65+, 66% of African Americans, and 61% of Hispanic Americans. 59% of high school grads, 46% of those with less than a high school education.

Part of the problem for vaccine prioritization is that, with imperfect knowledge, it is impossible to calculate risks accurately. The CDC says that persons with diabetes are high risk and should have priority for vaccination, but it also estimates that 1 in 5 persons with diabetes do not know they have it. Those persons, who are more likely to be in communities without medical care, would not even know they should be high-priority for a vaccine.

And there are undoubtedly many health conditions that may put persons at higher risk of severe COVID but this risk unknown for lack of data. For example, many Arizonans and Californians have had Valley Fever, a fungal lung infection that, in severe cases, can cause lung scarring. But Valley Fever is not as prevalent as conditions such as diabetes or asthma, and thus there just is not enough data to assess potential additional risk of COVID to persons with a history of Valley Fever.

References

Deming, W. Edwards (1986). Out of the Crisis. Cambridge, MA: MIT Press.

Deming, W. Edwards (1994). The New Economics for Industry, Government, Education. Cambridge, MA: MIT Press.

Deming, W. Edwards (2013). The Essential Deming: Leadership Principles from the Father of Quality. Edited by Joyce Orsini. New York: McGraw-Hill.