TESS Special Competition on Replications

Time-sharing Experiments for the Social Sciences (TESS) is an NSF-funded initiative. Investigators propose survey experiments to be fielded using a nationally representative US adults sample via NORC at the University of Chicago's probability-based AmeriSpeak® Panel.

We are excited to announce a Special Opportunity for researchers to propose survey experiments that replicate (and potentially extend) prior published work.

In terms of specifics:

• We define replication as involving the collection of new data (i.e., the study being proposed) using the same design/procedure/material/measures as a prior published experimental study. This is sometimes referred to as “repeatability.”


• Characteristics of appropriate prior experiments to propose replicating for this competition:


• The prior study must be published (or accepted for publication) via a peer-reviewed process (article, monograph, or book) in a social science discipline.


• The prior study must be a survey experiment.


• The prior study can, but need not be, a prior TESS study. Information on past TESS studies can be found at: https://www.tessexperiments.org/paststudies.


• The prior study could have been carried out in any context (e.g., country).


• The proposer does not need to be (but could be) the author of the prior study.


• Applicants must provide a citation and a link to the prior study and a roughly half page description of it as part of their proposal.


• If the proposed design/procedure/material/measures are not identical, the applicant should explain why in their proposal. (Useful guidance on replication is provided in Brandt et al. (2014), Journal of Experimental Social Psychology.


• The replication should not be a so-called “conceptual replication,” in which a deliberately different experimental design than the original study is used to test a common theoretical claim.


• Applicants are allowed (and in fact encouraged) to incorporate additional new elements into the design such as additional conditions, measures (pre- or post-treatment), and so on that expand the design. This may be to assess the generalizability of a prior experiment, as long as the original to-be-replicated design is still present (as part of the design) and not substantively altered.


• There is no requirement for the applicant to expect that the replication attempt will replicate effects or not. The proposal should explain not only why the replication is of interest but also why it may or may not replicate (or why it is unclear), and why that would be of interest. An ideal replication proposal would be one where an outcome of either an effect or a null finding is interesting.


• Given that even successful replications usually find smaller effect sizes than originally published studies, applicants are required to provide a power analysis. The analyses for the replication must have power of at least .9. If the proposed experiment includes conditions (extensions) beyond the original study, we suggest that the analyses relevant to the extension part of the experiment have power of at least roughly .8 (this is not a strict requirement, however). The power analysis is not counted toward the proposal page limit.


• To accommodate the higher power required for good replication attempts, applicants are allowed to request studies that are significantly larger than a typical TESS application in terms of sample size/unit parameters. The below table provides the allowable parameters for the special competition.


• If the proposal includes a special sample (e.g., because the original experiment did so), additional expenses as described at https://www.tessexperiments.org/html/limits.html apply. Such special samples include particular subgroups (including teen samples). For specifics, proposers can contact tess.experiments@gmail.com.


• Before being fielded, we will require an accepted applicant pre-register their experiment (at the pre-registration site of their choosing).


• Application expectations are otherwise the same for other TESS proposals, and the instructions and requirements for these are provided at http://www.tessexperiments.org. These details should be reviewed by anyone wishing to apply.


• The exception is that proposals can be up to six, rather than five, pages of text.


• Please make clear at the top of the submission that the proposal is for the special competition.


• TESS anticipates fielding 3-7 special competition projects via this mechanism.


• For any other questions, please contact tess.experiments@gmail.com. Also, as with all TESS proposals, investigators whose proposals are successful will subsequently need to obtain IRB approval from their home institutions before their experiment can be fielded.


• All blinded proposals (and summaries of them) will be made available, on a limited basis, for a forecasting elicitation aimed to assess whether scholars can predict which studies are accepted. The proposals will not be made public, however (other than TESS’s general policy of making accepted proposal’s public when data are publicly released).

The deadline for proposals is September 15, 2023.

The table below provides sample size and unit guidelines for the TESS Special Competition on Replications.

Units	General Population Sample Maximum N
10	5000
11	4900
12	4500
13	4150
14	3800
15	3600
16	3900
17	3700
18	3500
19	3350
20	3200
25	2800
30	2300
35	2000