Limitations

There are four criteria applicable to the assessment of any research: truth value, applicability, consistency, and neutrality.

How do we achieve these criteria in a quantitative study?

Truth value: internal validity – Can potential observed changes be attributed to the program or intervention (i.e., the cause) and not to other possible causes (sometimes described as "alternative explanations" for the outcome)? Threats to internal validity can be grouped in three general categories. The first involve the single group threats – criticisms that apply when there is only a single group that receives the treatment or program. Avoid it by using a comparison group. The second consists of the multiple group threats – criticisms that are likely to be raised when there are several groups in a study (e.g., a program and a comparison group). It is important to know if they are similar at the start. Avoid it by having a pretest/covariate. Finally, there are social threats to internal validity -- threats that arise because social research is conducted in real-world human contexts where people will react to not only what affects them, but also to what is happening to others around them. Possible social threats include: diffusion or imitation of treatment, compensatory rivalry, resentful demoralization, and compensatory equalization of treatment. Teachers implementing the treatment and/or comparison need to be familiar with the research process and can be offered training on the treatment and implementation of the treatment after the data for the study is collected.

Applicability: external validity – Is there any generalizability, or how is the study limited by the confines of a convenience sample? External validity is increased when more than one site is used.

Consistency: reliability – Are the instruments for collecting data and analyses appropriate and verified for repeated use?

Neutrality: objectivity – Does the research design take the greatest advantage of the distribution of treatment and comparison groups and a wide variety of subjects? Is the treatment being implemented fairly and not likely to cause an unnecessary bias? Is the  value for the study set a priori and appropriate, based on either literature or standards of the field?