Which type of validity represents the extent to which the results of a study can be generalized?

Indian J Psychol Med. 2018 Sep-Oct; 40(5): 498–499.

Abstract

Reliability and validity describe desirable psychometric characteristics of research instruments. The concept of validity is also applied to research studies and their findings. Internal validity examines whether the study design, conduct, and analysis answer the research questions without bias. External validity examines whether the study findings can be generalized to other contexts. Ecological validity examines, specifically, whether the study findings can be generalized to real-life settings; thus ecological validity is a subtype of external validity. These concepts are explained using examples so that readers may understand why the consideration of internal, external, and ecological validity is important for designing and conducting studies, and for understanding the merits of published research.

Key words: CATIE, ecological validity, external validity, internal validity, reliability

DID CATIE HAVE EXTERNAL VALIDITY?

The answer is both yes and no. CATIE[1] was designed as an effectiveness study; that is, a study with relevance to real-world settings. The CATIE findings are relevant to clinical practice in the USA but are of questionable relevance in India. One reason is that, in the USA, where CATIE was conducted, the primary outcome, time to all-cause treatment discontinuation, is substantially patient-influenced, whereas in India, where families supervise treatment, it is largely caregiver-determined. Another and more important reason is that the healthcare delivery system in clinical practice is strikingly different in the two countries. Thus CATIE has good external validity for clinical practice in the USA but not in India.

RELIABILITY AND VALIDITY

Reliability and validity are concepts that are applied to instruments such as rating scales and screening tools. Validity describes how well an instrument does what it is supposed to do. For example, does an instrument that screens for depression do so with high sensitivity and specificity? Reliability describes the consistency with which results are obtained. For example, if an instrument that rates the severity of depression is administered to the same patient twice within the span of an hour, are the scores obtained closely similar? Different types of reliability and validity describe desirable psychometric properties of research and clinical instruments.[2,3] Validity can also be applied to laboratory and clinical studies, and to their findings, as well, as the sections below show.

INTERNAL VALIDITY

Internal validity examines whether the manner in which a study was designed, conducted, and analyzed allows trustworthy answers to the research questions in the study. For example, improper randomization, inadvertent unblinding of patients or raters, excessive use of rescue medication, and missing data can all undermine the fidelity of the results and conclusions of a randomized controlled trial (RCT). That is, the internal validity of the RCT is compromised. Internal validity is based on judgment and is not a computed statistic.

Internal validity examines the extent to which systematic error (bias) is present. Such systematic error can arise through selection bias, performance bias, detection bias, and attrition bias.[4] If internal validity is compromised, it can occasionally be improved, for example, by a modified plan of analysis. However, biases can be often fatal as, for example, if double-blind ratings were not obtained in an RCT.

EXTERNAL VALIDITY

External validity examines whether the findings of a study can be generalized to other contexts.[4] Studies are conducted on samples, and if sampling was random, the sample is representative of the population, and so the results of a study can validly be generalized to the population from which the sample was drawn. But results may not be generalizable to other populations. Thus external validity is poor for studies with sociodemographic restrictions; studies that exclude severely ill and suicidal patients, or patients with personality disorders, substance use disorders, and medical comorbidities; studies that disallow concurrent treatments; and so on. External validity is also limited in short-term studies of patients who need to be treated for months to years. External validity, like internal validity, is based on judgment and is not a computed statistic.

ECOLOGICAL VALIDITY

Ecological validity examines whether the results of a study can be generalized to real-life settings.[5] How is this different from external validity? External validity asks whether the findings of a study can be generalized to patients with characteristics that are different from those in the study, or patients who are treated in a different way, or patients who are followed up for longer durations. In contrast, ecological validity specifically examines whether the findings of a study can be generalized to naturalistic situations, such as clinical practice in everyday life. Ecological validity is, therefore, a subtype of external validity. The ecological validity of an instrument can be computed as a correlation between ratings obtained with that instrument and an appropriate measure in naturalistic practice or in everyday life. The ecological validity of a study is a judgment and is not a computed statistic.

DISCUSSION

Ecological validity was originally invoked in the context of laboratory studies that required to be generalized to real-life situations.[5] Thus, laboratory studies of the neuropsychological and psychomotor impairments produced by psychotropic drugs have poor ecological validity because what is studied in relaxed, rested, and healthy subjects tested in a controlled environment is very different from demands that stressed patients face in everyday life. In fact, these cognitive and psychomotor tests, especially when based on computerized tasks, have no parallel in everyday life. How much less ecological validity, then, would research in animal models of different neuropsychiatric states have for patients in clinical practice? This explains why drugs that work in animal models often fail in humans.[6]

On a parting note, a good understanding of the concepts of internal, external, and ecological validity is necessary to properly design and conduct studies and to evaluate the merits and applications of published research.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

REFERENCES

1. Lieberman JA, Stroup TS. The NIMH-CATIE schizophrenia study: What did we learn? Am J Psychiatry. 2011;168:770–5. [PubMed] [Google Scholar]

2. Streiner DL. Statistics commentary series: Commentary #15-reliability. J Clin Psychopharmacol. 2016;36:305–7. [PubMed] [Google Scholar]

3. Streiner DL. Statistics commentary series: Commentary no 17-validity. J Clin Psychopharmacol. 2016;36:542–4. [PubMed] [Google Scholar]

4. Juni P, Altman DG, Egger M. Assessing the quality of randomized controlled trials. In: Egger M, Smith GD, Altman DG, editors. Systematic Reviews in Health Care: Meta-analysis in Context. London: BMJ Publishing Group; 2001. pp. 87–108. [Google Scholar]

5. Lewkowicz DJ. The concept of ecological validity: What are its limitations and is it bad to be invalid. Infancy. 2001;2:437–50. [Google Scholar]

6. Andrade C, Sudha S, Venkataraman BV. Herbal treatments for ECS-induced memory deficits: A review of research and a discussion on animal models. J ECT. 2000;16:144–56. [PubMed] [Google Scholar]

Articles from Indian Journal of Psychological Medicine are provided here courtesy of Indian Psychiatric Society South Zonal Branch