An Interviewer Error in the Iowa Poll: How Much Effect?
The last CNN/Des Moines Register/Mediacom Iowa poll before tonight’s Iowa caucuses was scheduled to be released on February 1, but the release was canceled. According to the New York Times, an Iowa supporter of Pete Buttigieg who was called by the poll reported that Buttigieg’s name was omitted from the list of candidates. The New York Times reported:
The poll is conducted by telephone from a call center, where operators read from a prepared script of candidates’ names to determine who a voter plans to support. One operator had apparently enlarged the font size on their computer screen, perhaps cutting off Mr. Buttigieg’s name from the list of options, according to two people familiar with the incident who did not have permission to speak about it publicly.
The pollsters stated that this error appeared to be limited to one survey interviewer, but could not confirm this with certainty, and so decided, out of an “abundance of caution,” to cancel the release of the poll. Of course, pollsters are always concerned about how their polls are perceived but, in reality, the interviewer’s error would not have biased the poll’s results.
The interviewer’s error would have created a bias in the poll if candidate Buttigieg’s name was left off the list for every poll respondent. But the Iowa Poll re-randomizes the order in which names are read before every interview.
Why do pollsters randomize the order in which candidate’s names are read? It’s because of how people make choices when presented with a list of items. Some will choose an early item on the list and tune out for the rest of the list; others may be more likely to select the last item read. By randomizing the order in which names are read, pollsters ensure that each candidate is affected equally by the list positioning. Some respondents hear Biden’s name read first in the list; others hear Warren’s name read first; others hear Klobuchar’s name read first, etc. Any advantages from being first or last in the list are spread equally among all the candidates.
The randomization also means that no candidate would have been disadvantaged, relative to the other candidates, by having one interviewer omit reading the last name in the randomized list. Even if every interviewer enlarged the font and omitted the last candidate’s name, that type of error would affect all the candidates equally. Each candidate’s name would be expected to be omitted for the same number of poll respondents. This particular error can be considered to be a random interviewer effect.
All polls conducted by interviewers have interviewer effects. Some respondents may give a different answer to a female interviewer than to a male interviewer; some may find one interviewer harder to understand than another interviewer; an interviewer might make an error while reading a question on the script to one respondent. These interviewer effects tend to average out over all the poll respondents and interviewers and thus increase the variability of the statistics, but not the bias. The same thing would happen for this case, where one interviewer inadvertently neglected to read the last candidate’s name.
Moreover, this type of error is likely quite small compared with other sources of error in the poll. The methodology report for an earlier Iowa Poll from January said that poll was “based on telephone interviews with 3,131 registered voters in Iowa, including 701 who say they will definitely or probably attend the 2020 Democratic caucuses.” The list of voters came from the Iowa Secretary of State’s voter registration list, and the sample “was supplemented with additional phone number lookups.” It is likely that the list has some undercoverage — that is, it does not include every Iowa caucus-goer. The list might be missing recent voter registrations, and for some of the registered voters no telephone number is available.
In addition, not everyone selected for a poll participates in it. How many calls were made to obtain interviews with 3,131 registered voters? How many persons were called but refused to talk to the pollster? The methodology report does not give the response rate for the poll, but typically fewer than 10 percent of the persons called in a poll provide answers to the pollster. Pollsters weight the data to try to compensate for nonresponse, but it is likely that the errors from nonresponse and undercoverage, and the sampling error (after accounting for the nonresponse weighting, a poll of about 700 persons would have a margin of error of about 5 percentage points) would be much, much larger than the trivial, non-biasing error from one font-enlarging interviewer.
Copyright (c) 2020 Sharon L. Lohr
Postscript added February 7, 2020: The data from the February 1 Des Moines Register poll would be even more valuable now, following the confusion in collecting and counting the results from the Iowa caucuses on February 3. The pollsters could delete the presbyopic interviewer’s data before calculating statistics, if desired (as argued above, even if other interviewers also enlarged the font, this would not bias the results). But the poll results would be helpful for studying last-minute trends in voter preferences as well as the caucus process itself. I hope the pollsters will decide to release the data, to provide information for today as well as for future social scientists and historians.