Equivalency Framing of Societal Problems and Policy Solutions

Download data and study materials from OSF

Principal investigators:

Laura Stoker

University of California, Berkeley

Email: stoker@berkeley.edu

Homepage: https://polisci.berkeley.edu/people/person/laura-stoker

Amy E. Lerman

University of California, Berkeley

Email: alerman@berkeley.edu

Homepage: https://amylerman.weebly.com/

Alexander Sahn

University of California, Santa Barbara

Email: asahn@berkeley.edu

Homepage: https://www.alexandersahn.com/


Sample size: 3510

Field period: 06/25/2019-03/15/2020

Abstract

Governments, interest groups, and the mass media routinely use quantitative information to depict the severity of problems facing the nation. When doing so, they can choose to report the (unacceptably low) percentage of people experiencing good outcomes—as in the percentage of high school students obtaining the degree—or the (unacceptably high) percentage of people experiencing bad outcomes—as in the percentage of high school students dropping out. Likewise, policy initiatives to improve matters can be framed as increasing the incidence of good outcomes or as decreasing the incidence of bad ones. Despite the ubiquity of these rhetorical choices in political discourse, there is no research into whether they matter to public opinion. Existing studies of negativity bias, loss aversion, and attribute framing do, however, support the expectation that they will matter.

We carried out two sets of experiments to open up this line of inquiry. Each set consists of four experiments focusing, respectively, on drinking water quality, hospital deserts, high school achievement, and marine debris. The first set presents problem information, randomizing whether it is framed negatively or positively, and the second presents a policy solution, again with the frame randomized. All four experiments within each set showed clear and consistent results. When problems were negatively (vs. positively) framed, people expressed more anger and worry, and judged the problem more serious and more worthy of government attention. The public’s reaction to policy solutions, by contrast, were utterly unaffected by the frame.

Hypotheses
Experimental Manipulations

In this study, respondents were given either a problem or a policy question. The problems and policies were assigned randomly from a list of 4, which included:

  1. Drinking Water Quality
  2. Hospital Deserts
  3. High School Achievement
  4. Marine Debris
For each problem, we randomly assigned respondents to one of two potential problem frames, either positive or negative. In the negative frame, respondents were told that the bad outcome was present at a rate of 20% while in the positive frame, respondents were told that the good outcome was present at a rate of 80; we only used problems where the 20%/80% breakdown was roughly accurate, and proved links to relevant studies in the debriefing materials. To illustrate, this is the wording Hospital Deserts treatments:

The chance of surviving a life-threatening medical emergency can depend on how long it takes to get to a hospital. According to a recent report, [20% / 80%] of Americans live [too far from / close enough to] a hospital to get adequate care in case of an emergency.

The policy experiment treatments first introduced the problem and then described a policy initiative designed to ameliorate it. In the negative condition, the problem was described in terms of the prevalence of bad outcomes (20%) and the policy was portrayed as reducing that from 20% to 10%. In the positive condition, the problem was described in terms of the prevalence of good outcomes (80%) and the policy was portrayed as increasing that from 80% to 90%. To illustrate, this is the text of the Hospital Deserts conditions.

The chance of surviving a life-threatening medical emergency can depend on how long it takes to get to a hospital. According to a recent report, [20%/80%] of Americans live [too far from/close enough to] a hospital to get adequate care in case of an emergency.

Representatives in Congress have proposed a policy that would decrease the number of Americans that live [too far from / close enough to] a hospital to get adequate care. The policy proposal calls for a coordinated effort across the 50 states, the creation of public-private partnerships, and a combination of tax incentives, regulatory reforms and outreach initiatives. The price tag is estimated to be $10 billion/year. The policy is expected to [decrease / increase] the percentage of Americans living [too far from / close enough to] hospitals from [20% to 10% / 80% to 90%] by 2025.

Overall, the design was a 4x2x2 factorial—questions regarding four topic areas, each framed either negatively or positively, and either focused on reactions to problems or reactions to policy solutions. Respondents were randomized into one of the 16 possible conditions.

Outcomes

Problem Dependent Variables

For each of the dependent variables shown below, the order of the response options was randomized (e.g. Not angry – Extremely angry/Extremely angry – Not angry). The order in which the four questions were asked was also randomized.

Anger
How angry does this situation make you? Worry
How worried are you about this problem? Serious
In your opinion, how serious of a problem is this situation? Priority
How much priority should the U.S. government give to improving this situation? Plus an 8th option always placed below others: “Government should not be involved in this area.”

Policy Dependent Variables

For each of the dependent variables shown below, the order of the response options was randomized (e.g. Strongly Disapprove to Strongly Approve vs. Strongly Approve to Strongly Disapprove). The order in which the questions were presented was fixed, as given below.

Opinion
What is your view of this policy proposal? Petition
Would you be willing to support a petition in support of this policy? Volunteer
One element of the outreach is to engage citizen volunteers in efforts to solve the problem. Would you be willing to lend your time and effort to this effort?

Other Variables

In addition to the core problem or policy DV questions, we also asked a standard set of demographics, plus questions concerning party identification. We also asked questions to measure two potential moderators: Risk Attitude and Need for Cognition. These questions were asked at the start of the survey, prior to the experiment. After the last survey question, we offered an optional but full debriefing. The debriefing page included study information as well as source information and links for each of the problem depictions.

Summary of Results

Problem Experiments
Results were in the expected direction across all four experiments and all four dependent variables (DVs). Negatively framed problems tended to generate more anger and worry, and to be judged more serious and more worthy of government attention. In 10 of the 16 comparisons (four issues, four DVs each), the treatment effect was substantively and statistically significant (p<.05, two-tailed). If the data for the four experiments were stacked and analyzed with fixed effects for problem area and DV order randomizations, treatment effects were significant for all four dependent variables at p<.001. However, the effect size was significantly larger for Anger (b=.66, with Y scored 0-1, beta=.17) than for Worry (b=.44, beta=.12), Seriousness (b=.44, beta=.13), and Government Priority (b=.26, beta=.08). There was some evidence of anger carry-over effects, in that the effect sizes for Worry, Seriousness, and Government Priority were larger if the respondent was first asked about the extent to which the problem made him or her angry. Effect sizes were not moderated by Need for Cognition or Risk Attitude. They were larger among college-educated respondents (vs. the less educated) and among political Independents (vs. among Democrats or Republicans), but these differences did not reach conventional levels of statistical significance.

Policy Experiments
Results were consistent across all four experiments and all three dependent variables. In no case was there a substantively significant difference in opinions by frame, and 11 of the 12 tests of framing effects (four issues, three DVs each) yielded a p-value greater than .40. Moreover, we found no evidence of significant effects among subgroups defined by potential moderators (Need for Cognition, Risk Atittude, Education, Age, or Partisanship).

References
Pilot Study Results from a survey of MTurk workers were reported in “Public Opinion on Policy Solutions—the Role of Equivalence Frames, Policy Scope, and Party Cues,” a paper presented at the 2018 American Political Science Association Convention. A paper reporting on both the TESS and MTurk studies will be under review soon.