Assessing the Prevalence of Socially Undesirable Opinions: The Use of the Double List Experiment for Variance Reduction and Diagnostics

Download data and study materials from OSF

Principal investigators:

Adam Berinsky

Massachusetts Institute of Technology

Email: berinsky@mit.edu

Homepage: http://web.mit.edu/berinsky/www/

Adam Glynn

Harvard University

Email: aglynn@fas.harvard.edu

Homepage: http://scholar.harvard.edu/aglynn


Sample size: 5550

Field period: 8/13/2009-12/28/2009

Abstract

A classic concern in survey research is how best to elicit information about support for socially sensitive policies or candidate choices. One solution to this problem is to unobtrusively measure support using a `list experiment'. The list experiment works by cloaking the sensitive `treatment' item among a list of other non-sensitive items. The list experiment allows a researcher to measure sensitive attitudes because, if respondents answer the sensitive question honestly (due to the privacy provided by the question format), the analyst can estimate the true proportion in the population that hold the sensitive opinion. This is often accomplished by taking the difference between the average response among the treatment group and the average response among the baseline group.

A great deal of work in political science and sociology, including work supported by TESS, has been devoted to the use of the list experiment. While this work is important, not enough attention has been paid to the particulars of the list experiment. Most work in political science has been done using a modified version of a list created by Gilens set al. for a 1991 survey. As list experiments become more widely used in political science, it is prudent to conduct research to lay out the "best practices" for such experiments. In this study, we examine best practices for the list experiment by performing two double list experiments, that allow us to assess the validity and efficacy of previous work with list experiments in political science.

Hypotheses

The power limitations with the current usage of the list experiment are due to two factors. First, by design, the single list experiment only presents the sensitive item to half of the respondents. Therefore, the sample size is effectively cut in half. Second, the baseline lists employed by political scientists are typically modified from the Gilens et al. list, and hence little is known about the power benefits to be derived from alternate baseline lists. The reduction in power due to these two factors makes it difficult to assess with a high degree of confidence, the proportion of people who are angered by a particular item within politically relevant subgroups of the population. Political scientists are often interested in making such comparisons. We employ a new design that allows the sensitive item to be addressed to all respondents, and that varies the correlation of the treatment items within and across two experimental lists – the "negative correlation within, positive correlation between double list experiment" design. This design provides a potential solution to both of the problems identified above. Our expectation was that the double list would provide greater statistical power than list experiments commonly used in political science.

Outcomes

Condition A
BG1A
1. Now I'm going to read you four things that could make some people angry or upset. After I read all four, just tell me HOW MANY of them upset you. I don't want to know which ones, just HOW MANY. <Note: randomize item list>
(1) The way gasoline prices keep going up.
(2) Professional athletes getting million dollar-plus salaries.
(3) Requiring seat belts to be used when driving.
(4) Large corporations polluting the environment.
BG2A
2. Now I'm going to read you five things that could make some people angry or upset. After I read all five, just tell me HOW MANY of them upset you. I don't want to know which ones, just HOW MANY. <Note: randomize item list>
(1) Increasing taxes on the wealthy
(2) Decreased spending on welfare programs
(3) The anti-gay marriage initiative in California
(4)  Organizations that support a woman’s right to have an abortion, such as Planned Parenthood
(5) A woman serving as president

Condition B
BG1B
1. Now I'm going to read you five things that could make some people angry or upset. After I read all five, just tell me HOW MANY of them upset you. I don't want to know which ones, just HOW MANY. <Note: randomize item list>
(1) The way gasoline prices keep going up.

(2) Professional athletes getting million dollar-plus salaries
(3) Requiring seat belts to be used when driving
(4) Large corporations polluting the environment
(5) A woman serving as president
BG2B
2. Now I'm going to read you four things that could make some people angry or upset. After I read all four, just tell me HOW MANY of them upset you. I don't want to know which ones, just HOW MANY.. <Note: randomize item list>
(1) Increasing taxes on the wealthy
(2) Decreased spending on welfare programs
(3) The anti-gay marriage initiative in California
(4)  Organizations that support a woman’s right to have an abortion, such as Planned Parenthood

Condition C

BG1C
1. Now I'm going to read you four things that could make some people angry or upset. After I read all four, just tell me HOW MANY of them upset you. I don't want to know which ones, just HOW MANY. <Note: randomize item list>
(1) Increasing the minimum wage
(2) Decreased spending on public health care
(3) Allowing prayer in public schools
(4) Having a baby outside of marriage

BG2C
2. Now I'm going to read you five things that could make some people angry or upset. After I read all five, just tell me HOW MANY of them upset you. I don't want to know which ones, just HOW MANY. <Note: randomize item list>
(1) Increasing taxes on the wealthy
(2) Decreased spending on welfare programs
(3) The anti-gay marriage initiative in California
(4)  Organizations that support a woman’s right to have an abortion, such as Planned Parenthood
(5) A woman serving as president

Condition D
BG1D
1. Now I'm going to read you five things that could make some people angry or upset. After I read all four, just tell me HOW MANY of them upset you. I don't want to know which ones, just HOW MANY.. <Note: randomize item list>
(1) Increasing the minimum wage
(2) Decreased spending on public health care
(3) Allowing prayer in public schools
(4) Having a baby outside of marriage
(5) A woman serving as president
BG2D
2. Now I'm going to read you four things that could make some people angry or upset. After I read all five, just tell me HOW MANY of them upset you. I don't want to know which ones, just HOW MANY. <Note: randomize item list>
(1) Increasing taxes on the wealthy
(2) Decreased spending on welfare programs
(3) The anti-gay marriage initiative in California
(4)  Organizations that support a woman’s right to have an abortion, such as Planned Parenthood

Summary of Results

In order to effectively employ the double list experiment, we needed to create two baseline lists that met our design goals. The canonical baseline list developed by Kuklinski et al. (1997) consists of a series of non-political attitude items. We began our work by constructing instead a list composed of political attitude items. We selected this strategy for two reasons. First, we believed that it would be easier to create items that would induce negative within-list correlation with political items than the items on the Kuklinski (1997) list. Second, we believed that the target item would not stand out as starkly when intermixed with attitude statements regarding political controversies. The exact items on these lists are presented above.

Many of the stated design objectives were achieved by these lists. First, the within list negative correlation design reduced the standard error of the difference-in-means estimator. Second, the between list positive correlation design reduced the standard error for both male and female respondents. For male respondents, using the double list approach reduces the standard error by a full percentage point (from 5.0 to 4.0). However, the lists did not perform exactly as expected.

First, there were a couple of negative estimates produced by the experiment. This is clearly nonsensical, and these results either indicate a lack of comparability between the treatment and control groups (due to an insufficient sample size), or they indicate that at least some respondents are misrepresenting their answers on the ``how many'' questions. Note that we cannot rule out the possibility that the negative numbers are due to sample size. Second, there was a bit more variability than we might expect between some of the estimates. In particular, we obtained estimates for female respondents between -3.3 and 12.5. This may indicate list order effects.