1. to provide social scientists with a new opportunity for original data collection and discovery
2. to promote innovative experimentation in social science
3. to increase the precision with which fundamental social, political and economic dynamics are measured and understood
4. to increase the speed and efficiency with which advances in social scientific theory and analyses can be applied to critical social problems
5. to maximize financial efficiency by combining otherwise separate studies, thereby radically reducing the average cost per study
6. to provide an online collection of exemplary experiments for teachers and students who want to learn more about social science experimentation

To achieve a representative sample, we contract with NORC, which conducts surveys using its AmeriSpeak Panel. AmeriSpeak is a nationally representative, probability-based panel based on NORC’s National Sample Frame, an area probability sample funded and managed by NORC and used for several NORC studies. These other studies include the General Social Survey funded by the National Science Foundation and the Survey of Consumer Finances sponsored by the U.S. Federal Reserve Board in cooperation with the U.S. Treasury Department. For the recruitment methodology, A summary of the AmeriSpeak Panel sample design used for the TESS projects can be accessed here.

Proposals may come from any substantive area within any discipline in the social sciences so long as they utilize experimental designs and seek to make a valuable contribution to knowledge. 

In the most common experimental design in social science, different subjects are randomly assigned to groups that receive some different stimulus, and then differences between groups in some outcome are assessed.  Designs that involve within-subject experimental manipulations are also acceptable. 

TESS does not consider proposals that involve presenting the same survey questions to all respondents and comparing non-assigned subgroups (for example, men vs. women, or lower vs. higher educated respondents).

Experimental research often involves testing specific hypotheses, but this is not required of TESS proposals. 

Jeremy Freese of Stanford University, James Druckman of University of Rochester, and Maureen Craig of Duke University are the current Principal Investigators. A multidisciplinary team of Associate PI's assists Freese, Druckman, and Craig in managing TESS. The team includes an accomplished and diverse roster of over 65 Associate PIs from across  the social sciences. Team members span several generations and multiple  disciplinary boundaries, and each member has established a reputation  in his or her respective field. Most importantly, they share our  enthusiasm for this project.

General population experiments are intended to combine the strengths of experimental and survey designs in supporting causal inferences in the social sciences.  Surveys provide an efficient and well-studied means of gathering descriptive information about populations.  There are, however, often substantial obstacles to drawing strong causal inferences from conventional survey data. Over the years, many have hoped that advances in statistical methods would help scholars use such survey data to "partial out" or control for all plausible rival interpretations of a potentially causal relationship. Despite massive advances in statistical methods over the years, few people are as optimistic today that statistics can solve all such causal inference problems. Instead, throughout the natural and social sciences employ experimental designs in order to combat the challenges of causal inference, experiments are widely regarded as the most decisive design for adjudicating competing hypotheses about what causes what.

Social science has conducted a broad variety of different kinds of laboratory experiments.  While these laboratory experiments provide strong tests of causal propositions, scientific audiences, policy makers, and the public sometimes request more than a causal demonstration.  In many cases, science and society benefit from knowing that our laboratory observations survive exposure to myriad conditions outside of the lab. Moreover, some critics have questioned the extent to which the usual subjects in social science experiments resemble broader, more diverse populations ).

General population experiments offer a powerful means for researchers to respond to such critiques.  They allow researchers to assign large subject populations to experimental conditions of their choosing.  TESS uses Internet surveys to engage randomly selected respondents with randomly selected stimuli. By moving the possibilities for experimentation outside of the laboratory in this way, we can strengthen the internal validity of social science research and interest a much broader group of social scientists in the possibilities of experimentation.

As such, general population experiments offer many advantages to social scientists. Laboratory experimenters, for example, can use general population experiments to show that observations generated in a laboratory can be replicated in very different conditions. They can also test new hypotheses that emerge from their work with smaller groups of subjects. Alternatively, investigators who use the internet to run experiments on "knowledge networks" (e.g., markets, which aggregate diffuse individual behavior into prices and social systems that aggregate many individual characteristics into social hierarchies) can reinforce their research agendas by using general population experiments to evaluate individual-level hypotheses that inevitably follow from network-level observations. Likewise, scholars can use general population experiments to clarify the causal implications of findings from conventional surveys. Additionally, for a wide range of scholars, general population experiments offer new opportunities to innovate. A special advantage of general population experiments is the broad and diverse subject pools that they allow researchers to contact.

Such experiments are particularly effective at documenting differences in the status of causal hypotheses between the type of people who are usually selected for laboratory experiments and those who are not. Though not all social scientists require large and diverse subject populations to accomplish their research goals, many do. Finally, TESS also provides opportunities to strengthen and improve a wide range of measurement issues. For example, those interested in how to assess race and ethnicity in an increasingly diverse society can use experimental methods to understand how the method of data collection affects response attributes. Moreover, although the philosophy of early survey research was to attempt to create a social vacuum in which people could express their "true" beliefs and opinions, the more recent acknowledgment of attitudes, beliefs and preferences as a function of both the person and situation has led to an interest in the systematic study of how context alters the opinions and preferences that are expressed.

A critical element of the TESS research strategy is multiple studies from different disciplines sharing common observational platforms, all exploiting the inferential power and measurement efficiencies of experimental designs. This time-sharing on data collection platforms is the key to economic efficiencies of TESS. By distributing the costs of sampling and fielding over a large number of studies, the marginal cost of each study can be greatly reduced.   The start-up costs for launching any kind of large data collection effort are substantial. Time-sharing allows social science to experience the considerable economies of scale that come from paying such start up costs only once.

In this respect, time-sharing follows an established scientific tradition. The natural sciences, for example, promote progress in many areas by instituting time-sharing on expensive instruments (e.g., particle accelerators and telescopes). Time-sharing of university computer resources allowed many researchers to experience the benefits of computation before PC's became feasible and cost effective, and it is still used for many especially computing-intensive scientific applications.

Our time-sharing strategy also promotes efficiency in data collection by collecting demographic information that all investigators can share. If the experiments were conducted independently, each investigator would use time to collect such data on their own - increasing redundancy and reducing overall efficiency.

TESS provides other economic efficiencies by reducing waste. Most national surveys collect responses from some relatively large, round number of respondents. Because most surveys are not designed to test a small number of specific hypotheses, there end up being more cases than was actually needed for some purposes, and perhaps too few for others. The flexibility of our instruments allow people to roll on for as many or few cases as they need, and they must justify the sample size requested as part of the proposal. Flexibility in the instruments will allow more investigators to be served more efficiently.

All proposals must be submitted through our proposal handling system.

Each proposal must also designate a contact author in the submission system. The contact author must be listed first on all proposal documents and is the person to whom all official TESS corresponds will be held.

Proposals are limited to five pages of text (including footnotes/endnotes), plus references, up to two pages of tables, and the actual survey items to be included.

Power analyses, which are encouraged, may also be referenced in the main text and placed in a short appendix that does not count against the five page limit. When revisions are invited to proposals, these may be accompanied by a memo that details changes; concision here is encouraged. The entire proposal with all appendices or supplements of any sort may not exceed twenty pages under any circumstances.

Proposals must be double-spaced and 12 point font. A smaller font is allowed in footnotes and endnotes, and there are not specific formatting requirements for tables.

Proposals that exceed these limits will be returned by TESS staff.

We seek proposals that break new ground in the hypotheses they investigate, the procedures they employ, or both.

The key to TESS success is to win over reviewers in your chosen field. Ideally, your proposal should offer the potential for a clear scientific advance whose relevance expands beyond any one discipline.

Proposals that report trial runs of novel and focal ideas will be viewed as more credible.

While not required, it is desirable if the proposal is conducted in coordination with non-TESS data collection endeavors, such as traditional laboratory experiments or field work.

No. See the introduction for an elaboration of what we mean by an experimental design. (As of January 1, 2022, TESS no longer accepts quasi-experiments.)

The "size" of a TESS experiment is a function of both the length of the experiment and the number of respondents (N). The shorter the experiment, the more respondents on which it can be conducted. This page provides the maximum N for studies of different length, as well as guidelines for how study length is calculated. Note that experiments that involve subsampling will involve some % reduction in the maximum N.

This allotment does not include the demographic and socioeconomic data that TESS provides for all studies. The standard delivery includes:

More information on the precise measurements provided for family income is available here.

To submit a proposal to TESS's Short Studies Program please follow the instructions provided here.

TESS provides a free service to investigators whose proposals are endorsed by the external reviewers and relevant Associate PI's. They must also meet standard human subjects requirements and be consistent with the requirements of the NORC AmeriSpeak platform, and they cannot place an extraordinary burden on TESS resources. That TESS is a collective endeavor implies that there are strict limits on what services we can provide to any one investigator.

The resource limits stated herein are real. Proposals are more likely to succeed, both in the review process and then once out in the field, if these limits are strictly interpreted.

A list of the items included for free in the TESS data delivery from NORC is provided here.

TESS is conducted in the course of NORC AmeriSpeak Panel surveys, which also include sets of "profile" variables. Investigators in TESS studies may add profile variables to their studies, which are counted as just a fraction of an original survey item. The profile variables expected to be of most interest to TESS investigators are those on the Public Affairs Profile, the Health Profile, and the Financial Profile.

Maybe! The platform can be used to do this for information at various geographic levels, down even to the census tract.  However, this must be done in a way that the information can still be included as data that TESS can release publicly following the embargo period.  For any geographic information below the state level, the information given to a respondent cannot be so specific that it might identify their area uniquely.  For example, if an experimental stimulus was to include information about the racial composition of a census tract, the information that the tract was "56.45% White" might identify it uniquely, as opposed to "55-60% White."

No, you don’t need your own consent form because AmeriSpeak Panel members give their consent to participate in AmeriSpeak projects when the join the panel. If your IRB expects you to include a consent statement, share the Amerispeak IRB document with them, and specifically point them to Appendices A-D. Appendix A is a screenshot of the web page for panelists who join online; the page has links to the Privacy Policy and Terms and Conditions toward the bottom. If the screenshot is too blurry to read, use this link to the actual page. Appendix B is the consent statement read aloud to panel members who join by telephone. Appendix C is the AmeriSpeak Privacy Policy, which includes many of the protections IRBs value; it is available online and is also emailed or mailed to panel members once they join. Appendix D is the AmeriSpeak Terms and Conditions.

Proposals should be e-mailed to tess.experiments@gmail.com, with the subject line “Short Study Proposal”. Proposals for this Program should not be submitted through the TESS Manuscript Central site, and any proposals submitted through the Manuscript Central site will be treated as regular TESS proposals, not SSP proposals.

Short study proposals must be 1-2 pages in length (single-spaced), plus the proposed items and any references included at the end as a separate page.  The experimental design of the study must be clear.  All proposals should include a title for the project on the first page, but no identifying information of the author.  Other information permitted as appendices to regular TESS proposals may also be appended to a Short Study proposal.  Proposals should be in PDF format.

Yes. See here for an elaboration of what we mean by an experimental design.

Investigators for all TESS studies must receive appropriate approval from their institutional IRBs before we field their study.  Typically, investigators wait until after their study has been approved for fielding before pursuing IRB approval.

IRB support information from NORC

While regular TESS proposals are externally reviewed, for purposes of expediency, we will typically review these projects internally -- that is, by the Principal Investigators and/or Associate Principal Investigators. Feedback on proposals apart from the decision likewise should be expected to be brief and quick.

For regular TESS proposals, the ultimate arbiter of length for any TESS study are length determinations made by our survey vendor (NORC) for pricing studies. For purposes here, a stimulus will be considered one item unless its estimated time of administration is longer than 45 seconds, in which case it will be counted as more than one item. A stimulus that requires more than 90 seconds to administer needs to be submitted using the regular TESS mechanism rather than the Short Study Program.

For ease of fielding, the Short Studies Program is for projects that involve no special subsampling.  If you have a short study that requires subsampling, you should submit a regular proposal using our regular mechanism.

Upon receipt, we ask an appropriate Associate PI to provide us with the e-mail addresses of two prospective reviewers. The Associate PI is chosen from the proposer's home discipline. For co-authored proposals, we will choose an Associate PI from the home discipline of the first named author. We contact reviewers electronically, attaching preformatted request letters outlining the criteria for review. Completed reviews are received electronically by the co-PIs, and shared electronically with the appropriate Associate PI to obtain his or her advice about whether the project is both worthy and ready to earn a place in the study.

Our review process is double-blind. The two referees will review proposals without knowledge of the identity of potential user, and the reviews will be returned to the potential users stripped of their identities as well. The Associate PIs and co-PIs will know the identities of potential users so that they can use that information to make sure all proposals receive fair consideration. The PIs will take care of conveying the good or bad news back to the potential users.

In some circumstances, we also may ask for additional pre-testing of proposed projects. This includes sometimes offering to work with proposers to pre-test measures using Amazon Mechanical Turk in cases where the PIs deem it desirable to evaluate or strengthen a project.

In sum, our organizational goal is to minimize both paperwork and the amount of time between when a good research design reaches us and when the experimental data are in investigators' hands.

For researchers, TESS is free. As a result, the number of requests for our services are typically far greater than the number of opportunities we can supply. Among the consequences of this:

1. Our ability to provide opportunities for so many people depends on the internal economies of scale generated by having researchers share data collection instruments. We are open to a wide range of innovative ideas, but are constrained financially in what we can do regarding any single proposal. For proposals that go beyond our basic offering we require a higher review standard.  In addition, for options that impose unusual costs (such as streaming video presentations), we may make acceptance contingent on the proposer agreeing to pay the additional costs.

2. In the event that the number of proposals received overwhelms the administrative capacity that our funding allows, we may find it necessary to revise the requirements for proposal submission. So please be sure to check this page for updates before submitting future proposals.

When the review process results in a positive judgment for a proposal, we say that it is provisionally accepted. After TESS makes such a judgment, we notify the contact author with a letter of congratulations.

Provisional acceptance is necessary but not sufficient for data collection to begin. Two additional steps must be taken: the proposal must obtain human subjects approval and it must have a successful trial run.

As with any empirical research done by faculty or graduate students, TESS cannot place an experiment on a data collection instrument, until it has in hand a document indicating that the project has received human subjects approval from a university human subjects committee or internal review board (henceforth, we use the term IRB to refer to such administrative bodies.)

This documentation must be sent to TESS electronically to us at tess.experiments@gmail.com.

There are no exceptions to this rule. There are also no circumstances in which TESS or its data collecting subcontractors can provide IRB reviews. IRB approval is the sole responsibility of the investigators.

We encourage investigators to obtain IRB approval as soon as possible. The longer a researcher waits to get such approval, the longer will be the delay in collecting their data.

NORC provides extensive information for use when applying for IRB approval of studies conducted using its panel (including TESS experiments).

If your project is deemed non-exempt from review by your institution’s IRB, a representative of your institution will need to sign an Inter-Institutional Agreement (IIA) with NORC that establishes your institution as the reviewer of record. NORC will provide a template agreement that you are welcome to use.

To repeat, if a proposal is accepted for placement on a TESS data collection instrument and if the investigators cannot obtain IRB approval, then it is ineligible for inclusion on a TESS data collection instrument.

You will be asked to submit a project description that can be used to obtain pricing information from NORC and that can be used to program the instrument. It is very important that this project description provide not only the questions but a clear description of the randomizations used and which groups of respondents receive which questions. The basic template for the needed project specification is here. The project specification should be provided in Microsoft Word or other standard word processor format (not as .pdf, as this is harder for programmers to work with).

With the order of final acceptance as their guide, the co-PIs will exercise discretion in combining individual investigators' modules into a given version of the Internet data collection efforts. The TESS PI's and data collection subcontractors will work closely with individual scholars to make sure their modules are appropriately situated within larger instruments.

For programming an instrument, multimedia materials will need to be provided to NORC in accordance with their specifications. Please provide a file with the best image quality and least compression possible. Preferred file types are as follow:

Video: .mp4 or .ogg

Audio: .wav or .mp3

Image: .jpg, .gif , tif, .pct, .psd, .tga, .eps, or .ai

You will have exclusive access to the data for one year after the data are delivered. After that time, the data will become available to others via TESS's partnership with the Open Science Foundation. Our goal is to encourage you, the original researchers, to analyze, present, and publish your results as quickly as possible while making innovative data available to the larger scientific community.

Researchers sometimes augment TESS data collection by providing supplemental funds from other sources. TESS policy is that all data from studies for which it provides any funding are posted, regardless of the fraction covered by TESS and the fraction covered by other sources.

All publications using TESS should include the following citation: "Data collected by Time-sharing Experiments for the Social Sciences, NSF Grant 0818839, Jeremy Freese and James Druckman, Principal Investigators."

Investigators must also agree to prepare a summary description of their study and results. This summary must include a description of the treatment effects. Investigators will first be asked to provide a summary at the end of the one-year period in which the data are embargoed from other users. We recognize that investigators may not always be ready to provide a summary at that point; we do, however, require that all investigators provide the summary within three years after their project was accepted. The summary information is then posted on our website.

To increase the speed at which new knowledge is made broadly available, we also require the investigators to notify us of any working papers or publications based on the experiments.