Impact of Response Scale Direction on Survey Responses to Factual/Behavioral Questions

Download data and study materials from OSF

Principal investigator:

Florian Keusch

University of Mannheim

Email: f.keusch@uni-mannheim.de

Homepage: http://sswml.uni-mannheim.de/Team/Florian%20Keusch/


Sample size: 1729

Field period: 09/13/2013-12/13/2013

Abstract

The existing research studying the impact of scale direction suffers from several weaknesses. First and foremost, the majority of the literature on scale direction has focused on attitudinal questions. To advance research in this topic, this study examines the impact of scale presentation order on answers to frequency/behavioral questions. The second weakness of the literature on scale direction lies in the lack of attention to other scale features that could moderate the impact of scale order. This experiment varied two additional features of the response scales, that is alignment of the scale (horizontal vs. vertical) and vagueness of verbal scale labels (vague vs. precise vs. vague+precise).

Hypotheses

The main research question of this study pertains to the influence of scale direction on survey responses to frequency/behavioral questions: Holding other features of a response scale constant, does the direction of the scale affects survey responses to frequency/behavioral questions, and, if yes, in what way? Previous research on attitudinal questions suggests that there is a tendency for responses to lean towards the starting point of a scale, regardless of the direction.

Concerning the interaction between scale direction and scale alignment, a larger order effect for the vertical presentation of the frequency scale than for the horizontal presentation is expected, i.e. a higher endorsement of the first scale points when the scale is presented vertically compared with horizontally.

Presumably, scales with precise labels will make the tasks of interpreting scale labels and of mapping to individual scale point more straightforward. As a result, scales with precise scale labels should be less prone to scale order effect. Accordingly, frequency scales using vague quantifiers are hypothesized to show a bigger scale presentation order effect and that precise verbal labels will reduce the influence of scale presentation order.

Finally, the experiment helps to evaluate whether scale direction effects differ by respondent characteristics (e.g., education, age, speeding).

Experimental Manipulations

The methodological experiment comprises two sets of questions: (1) six items measuring mental health modeled after items from the National Health Interview Survey (NHIS) and (2) four health related behavioral items, both using a 5-point fully-labeled unipolar frequency scale. A full factorial experiment crosses three factors for both sets of items. The first factor varies the direction of response scales. For half of the sample, the scale runs from high to low (e.g., "all of the time" – "most of the time" – "some of the time" – "a little of the time" – "never"). For the other half, the scale runs from low to high (e.g., "never" – "a little of the time" – "some of the time" – "most of the time" – "all of the time"). The second factor varies the presentation of the response scale on a screen. The response scale were presented either horizontally or vertically. The third factor varies the vagueness of scale verbal labels, creating three versions. One version employs only vague quantifiers (e.g., "all of the time" – "most of the time" – "some of the time" – "a little of the time" – "never"). The second version shows more precise frequency labels (e.g., "7 days" – "5 or 6 days" – "3 or 4 days" – "1 or 2 days" – "0 days"). In the third version, a combination of both labels was used (e.g., "all of the time (7 days)" – "most of the time (5 or 6 days)" – "some of the time (3 or 4 days)" – "a little of the time (1 or 2 days)" – "never (0 days)").

After a set of 15 filler questions, a set of 10 open-ended frequency questions ask for a numeric report ("During an average week, how often do you …?") to the same domains of mental health and health behaviors. The answers to these questions are used as validation criteria for the experimental questions. In additional, GfK profile variables (self-rated health and whether the respondents has ever been diagnosed with depression) are used as validation criteria.

Outcomes

The key outcome variable for this experiment are responses (i.e., distributions, means, and variances) to the 10 individual questions on mental health and health behaviors under the different experimental conditions, controlling for indicators of satisficing behavior. In additional, the study analyzes correlations between responses to the experimental questions and validation criteria (open-ended questions and GfK profile variables).

Summary of Results

Other than expected, bivariate analysis shows that scale direction and scale alignment had no significant effect on response distribution, means, and variance. However, respondents who received the precise scale labels were more likely to use the negative end point of the scale ("0 days") than respondents in the vague scale labels group ("never"). These differences in response distribution also led to a significant difference in means between these two experimental groups. These results are confirmed in multivariate models showing no significant interaction between the individual experimental variables as well as between the experimental variables and indicators of satisficing (i.e., speeding, age, and education). Correlations with answers to open-ended numeric frequency questions and external criteria reveal no advantage of one scale direction condition or one scale alignment condition over the other.

References

References