Beware this Cognitive Bias when Interpreting COVID-19 Statistics

Dr. Dulcamara enters the stage in a gold carriage accompanied by trumpet fanfare. The self-proclaimed “master of all things medical, with knowledge encyclopedical” is selling a magical elixir that cures paralytics, asthmatics, hysterics, apopletics and diabetics; removes wrinkles and smooths skin; ends senility and restores virility; and also destroys mice and bedbugs.

“How much is a bottle?” Dulcamara asks the crowd after describing these wonders. “One hundred crowns? … thirty? … twenty?” No! Because of the regard he holds for his countrymen, he will practically give it to them for just one crown!

You may be wondering what Donizetti’s opera “The Elixir of Love” has to do with statistics about COVID-19 (other than the fact that The Metropolitan Opera has streamed broadcasts of it while forced to close during the pandemic). In both, characters take advantage of a cognitive bias known as the anchoring effect to shape opinion. Anchoring refers to people’s tendency to give undue weight to the first piece of information received about a topic. The first number thrown out—relevant or not— “anchors” subsequent thoughts about the value.

You’ve probably used anchoring at some point to nudge others toward a preferred outcome. When negotiating the salary for a new job, you might casually mention that a colleague was just offered a new position paying $94,000. Or you might set a list price for your house that is $40,000 higher than its worth; the house appears more valuable to buyers if the listing price is high rather than low. Dr. Dulcamara, a master at anchoring, throws out an absurdly expensive initial price for his elixir (which was really cheap Bordeaux) so that the still absurdly expensive final price sounds reasonable by comparison.

Why is the anchoring effect considered a cognitive bias? Because it leads people to make decisions that are not entirely guided by rational considerations. Numerous experiments have shown that anchoring affects perceptions about statistics even when people know that the anchor has nothing to do with the topic at hand (see Daniel Kahneman’s wonderful book about cognitive biases, Thinking Fast and Slow).

In one famous experiment, 52 lawyers were asked to assume the role of a judge deciding the sentence for a fictitious shoplifting case. Each was asked to roll a pair of dice and record the sum as the prosecutor’s recommended sentence. About half, selected randomly, were given a pair of loaded dice that always gave the sum of 3; the other half were given dice that always gave the sum of 9. The participants knew that the number from the dice roll had nothing to do with the shoplifting case, yet those who rolled a “9” assigned a sentence that was 2.5 months longer, on average, than those who rolled a “3”.

Many anchors have been proffered in the efforts to influence public opinion about the response to the COVID-19 pandemic. Early in the pandemic, the number of U.S. COVID deaths was compared with annual deaths from influenza and traffic accidents, but the number of U.S. COVID deaths passed the annual number of traffic deaths (typically about 35,000 per year) in April and the typical annual number of influenza deaths (about 55,000 per year) in early May. As deaths have climbed, the proffered anchors have climbed too, as described by Philip Bump of the Washington Post. It has been claimed that the administration’s actions “saved 3 to 4 million lives” and that the number of deaths ”could have been billions of people if we had not done what we did.”

The key feature about these anchors is that they are not based on data; their sole purpose is to shape public perception. They are, in a sense, like the dice thrown in the experiment about sentencing or like the absurd price given by Dr. Dulcamara.

How do you reduce the effect of arbitrary anchors? With reliable data.

Reference Distributions and COVID Statistics

Figure 1. Reference distribution for height of U.S. adult men. Tom’s height is shown by the blue dot.

Suppose someone tells you that Tom, an adult male in the U.S., is five feet nine inches tall. Is Tom tall or short?

To answer this, you might imagine a reference distribution similar to that in Figure 1. This normal distribution with mean 69.2 inches and standard deviation 3 inches closely follows the distribution of height measurements from the National Health and Nutrition Examination Survey, a nationally representative sample from the U.S. population. The survey-takers follow a uniform procedure to measure survey participants’ heights, thus reducing errors from variable measurement techniques (and errors are reduced even more because height is measured, not self-reported, since people tend to exaggerate slightly when asked their height). Tom is close to the center of this distribution, and you would conclude from this reference distribution that he is close to average height.

Figure 2. Number of deaths from COVID-19 per 100,000 population for 57 countries, Jan 22 - Sep 1, 2020. The blue rectangle shows the U.S. position.

We do not have reference data of similar quality for COVID-19. There are no data from previous years for this new disease. Data from 2020 are of varying quality because methods for collecting statistics are not uniform; case and hospitalization statistics may reflect the availability of testing or medical care more than the prevalence of the disease. Probably the most accurate (or least inaccurate) indicator is number of deaths (or, rather deaths per 100,000 population, to standardize across areas) although, as I’ll argue below, those statistics also come with caveats.

Figure 2 displays a histogram of the number of deaths per 100,000 population attributed to COVID-19 for the 57 countries listed in The Economist’s daily economic and financial indicators.* The U.S. is at about the 86th percentile of this set of countries.

Figure 3 shows the statistics for the 57 countries ordered from largest to smallest. My purpose is not to assign rankings, but just to look at the distribution of reported deaths among the set of countries.**

Figure 3. Deaths per 100,000 population for 57 countries.

One caveat about the data in Figure 3 is that COVID-19 deaths, although more reliable indicators than hospitalizations or cases, are counted differently across countries. Belgium, for example, may be near the top of Figure 3 because, according to NPR, “authorities decided to be radically transparent, if perhaps a bit speculative, about the toll from the novel coronavirus. They include not only deaths that are confirmed to be virus-related, but even those suspected of being linked, whether the victim was tested or not.”

It is quite likely that other countries’ population mortality rates would be much higher if they counted deaths the same way as Belgium, and the relative positions would change. For most countries, the COVID-19 deaths displayed in Figures 2 and 3 likely understate the true number of deaths attributable to the disease. Some countries exclude persons who likely died from COVID-19 but were not tested for it; others may have reporting lags that lead to lower counts.

Many epidemiologists and statisticians recommend looking at excess deaths to obtain an alternative picture. Excess deaths are computed by comparing the number of deaths in a time period from 2020 with those from the same time period in the previous year (or the average for the time period from the previous three or five years). Excess death statistics show how 2020 mortality patterns differ from previous years.

The Economist and The New York Times display graphs of excess deaths for some countries (unfortunately, not the full set of 57 countries in Figure 3) and all states. I’m not going to reproduce those graphs here, but I urge you to look at them. For most countries, the excess deaths from all causes are larger (and would be expected to be larger) than those attributed to COVID-19

The CDC estimated 232,288 excess deaths from all causes in the U.S. during 2020 (up through the week ending August 22) when compared with prior years. This is about 32% larger than the “official” number of deaths, 175,651, attributed to COVID-19 as of August 22. Alaska and Hawaii are the only states with no excess deaths in 2020 when compared to previous years.

For Belgium, however, which counted all suspected COVID-19 deaths in the official counts, the reported “official count” of 9,690 deaths from COVID-19 was actually higher than the 7,941 excess deaths. When comparing excess death rates, the U.S. (70 per 100,000 population as of August 22) and Belgium (69 per 100,000 population) are quite similar.

Anchors and Reference Distributions

At the end of The Elixir of Love, tenor Nemorino, who traded his entire savings for a bottle of cheap red wine, has won his lady love and inherited a fortune from his rich uncle. No one is more surprised than Dr. Dulcamara at these outcomes. But his surprise does not stop him from promptly taking credit and selling out his remaining bottles of elixir. Undoubtedly his future sales pitches will prominently feature Nemorino—and neglect to mention the many, many elixir purchasers whose only return on investment was a hangover. Dr. Dulcamara’s anchor price substitutes for data in the villagers’ decisions to purchase.

A model from Imperial College published in March 2020 predicted 81% of the U.S. population would be infected and 2.2 million U.S. residents would die in a worst-case scenario. But this prediction, based on the limited data available at that time, assumed that absolutely no measures would be taken by society or individuals to reduce the spread of the virus. And even this worst-case scenario prediction for number of fatalities, which received overwhelming media attention partly because it was so much higher than worst-case predictions from other researchers’ models, was lower than the 3-4 million (or billions!) of “lives saved” put forward to shape opinion. These anchors have no more basis than Dr. Dulcamara’s 100-crown elixir valuation.

Figures 2 and 3, displaying countries’ reported COVID-19 population mortality rates, present an imperfect reference distribution. They do, however, show the general shape of the distribution of population mortality rates among this set of countries and, along with similar graphs displaying the distributions of excess deaths, show the wide variability in countries’ mortality outcomes. The distributions are highly skewed, with most countries exhibiting much lower population mortality rates than the United States. We know that it was possible to have better outcomes, because countries as varied as South Korea, Iceland, and Slovakia attained them.

Footnotes

*The histogram in Figure 2 will, of course, change if a different set of countries is used. I chose to use The Economist’s list because it reflects a worldwide distribution that includes all countries in the G20, the OECD, and the European Union. Other choices of a reference set are possible, as long as the choice is made independently of the COVID-19 statistics. It’s not fair to construct a histogram from countries that are chosen specifically because their COVID-19 statistics are higher than, or lower than, the U.S.

I downloaded the COVID-19 deaths per 100,000 population statistics from The New York Times on September 2, 2020.

The statistics in Figures 2 and 3 are population mortality rates, not case fatality rates. I wrote about the difference between these two measures in my first post about coronavirus. Population mortality rates are deaths per 100,000 persons in the population; case fatality rates are deaths per 100,000 persons who have been diagnosed with the disease. Population mortality rates are a much better choice for comparisons because the denominator of population is known accurately; case fatality rates are much more uncertain because the denominator of diagnosed cases will be higher in countries with well-developed testing regimens than in countries that do little testing. It’s also better to construct reference sets using population mortality rates rather than total deaths (without any denominator), because of course one would expect more fatalities from larger countries.

**See my book Measuring Crime: Behind the Statistics for a discussion of why cities should not be ranked based on their crime statistics. The same principles apply here.

References

Ariely, Dan (2010). Predictably Irrational: The Hidden Forces that Influence our Decisions. New York: HarperCollins.

Englich, Birte, Thomas Mussweiler, and Fritz Strack (2006). Playing dice with criminal sentences: The influence of irrelevant anchors on experts’ judicial decision making. Personality and Social Psychology Bulletin 32, 188-200.

Kahneman, Daniel (2011). Thinking Fast and Slow. New York: Farrar, Straus and Giroux.