Download data and study materials from OSF
Principal investigator:
Hanna Grol-Prokopczyk
University at Buffalo
Email: hgrol@buffalo.edu
Homepage: http://sociology.buffalo.edu/faculty_staff/faculty/grol-prokopczyk/
Sample size: 1771
Field period: 1/26/2010-6/24/2010
Anchoring vignettes are an increasingly popular tool for identifying and correcting for group differences in use of subjective ordered response categories. However, existing techniques to maximize response consistency (use of the same standards for self-ratings as for vignette-ratings), which center on matching vignette characters' demographic characteristics to respondents' characteristics, appear at times to be ineffective or to pose interpretive difficulties. Specifically, respondents often appear to neglect instructions to treat vignette characters as age peers. Furthermore, when vignette characters' sex is matched to respondents' sex, interpretation of sex differences in rating style is rendered problematic. This study applies two experimental manipulations to a national sample (n=1,765) to clarify best practices for enhancing response consistency. First, an analysis of two methods of highlighting vignette characters' age suggests that both yield better response consistency than previous, less prominent means. Second, a comparison of ratings of same- and opposite-sex vignette characters suggests that, with avoidable exceptions, the sex of the respondent rather than of the vignette character drives observed sex differences in rating style. Implications for design and interpretation of anchoring vignette studies are discussed. Findings also show significant sex, educational, and racial/ethnic differences in styles of rating health, and racial/ethnic differences in styles of rating political efficacy, underscoring the incomparability of unadjusted self-ratings.
1) Can improved instructions and/or anchoring vignette wording overcome previously observed violations of age-related response consistency?
2) Does respondent's sex or vignette character's sex (or both) drive observed sex differences in ratings of anchoring vignettes?
3) Do respondents' other demographic characteristics (e.g., education, race/ethnicity) predict differences in use of response categories when rating anchoring vignettes?
Half of respondents received anchoring vignettes with male names, and half with female names.
Furthermore, half of respondents received vignettes specifying each character's exact age, where this was the multiple of 5 nearest the respondent's own age; half received vignettes where characters' age was suggested implicitly in opening instructions (e.g., "What follows are descriptions of the health of some people your age").
Ratings of anchoring vignettes: 4 general health vignettes, and 3 political efficacy vignettes.
1) Improved instructions and anchoring vignette wording *do* appear to overcome previously observed violations of age-related response consistency.
2) With occasional and avoidable exceptions, respondents' sex rather than vignette characters' sex underlies sex differences in ratings of anchoring vignettes. Implications for design and interpretation of anchoring vignette studies are discussed.
3) Non-trivial differences in use of response categories by key demographic characteristics (especially race/ethnicity, education, and sex) were observed. This underscores the incomparability of unadjusted subjective self-ratings, and supports the need for survey tools such as anchoring vignettes to adjust for such reporting heterogeneity.