Panel Conditioning in Longitudinal Surveys: An Experiment
Sample size: 7600
Field period: 04/19/2011-11/02/2012
Researchers who create and/or analyze data from longitudinal surveys nearly always assume that respondents’ attributes are not changed as a result of being measured. Yet research in psychology, political science, and elsewhere suggests that this assumption is not always warranted. Using an experiment embedded within a well-known web-based survey, we assess (1) the magnitude of panel conditioning on survey questions about illicit behaviors and (2) the degree to which panel conditioning biases depend on the length of time between survey waves. We find evidence that answering questions about theft and drunk driving affects subsequent responses to those same questions, but only when baseline and follow-up surveys are relatively close together in time. We conclude by discussing the implications of our findings for survey design and for social scientists who use longitudinal data.
Hypothesis 1: Respondents’ behaviors will appear to change over time when the experience of being surveyed activates previously forgotten or faintly held memories.
Hypothesis 2: Respondents’ behaviors will (appear to) change over time when surveys ask about socially non-normative or stigmatized topics.
Hypothesis 3: Panel conditioning is more likely to occur when surveys are spaced more closely together in time.
Our experiment began with a random subset of 7,654 members of the larger KnowledgePanel sample. In September of 2011, these respondents completed a short online survey that included a special module of questions. The module asked respondents two randomly selected questions (from a list of four possible items) concerning their lifetime experiences with marijuana use, petty theft, drunk driving, and arrest; the ordering of the questions was also randomized so as to avoid potential sequencing effects. Respondents to the baseline survey were randomly assigned to be re-interviewed either one month or one year after the baseline interview (but not both). In October 2011, those selected for the one-month follow-up (n = 2,852) were invited to complete a survey that included all four questions from the first module (again, in random order). In September 2012, those selected for the one-year follow-up (n = 4,802) were asked to complete a survey that likewise asked all four questions (also in random order). In the absence of panel conditioning, we would not expect to see differences between ballot groups with respect to any of the focal items. We examine this possibility separately for each of our two follow-up groups.
Lifetime experiences with arrest, drunk driving, petty theft, and marijuana use (as observed in respondents' follow-up interview).
Our results suggest that survey spacing and memory activation/reflection play a key role in determining whether (and in what ways) respondents alter their reports from one survey wave to the next. After the baseline interview, treated respondents (i.e., respondents who had prior exposure to the focal item) were more likely to say that they had previously driven drunk or stolen something of little value (behaviors that may have been difficult to recall during their initial interview), but only when the lag between surveys was limited to one month. Similar effects were not observed for arrests or marijuana use.