Reliability is the consistency of a measurement procedure when the phenomenon being measured has not changed. A reliable measurement produces scores that are relatively free from measurement error. Validity refers to the extent to which a measurement actually measures the concept it is intended to measure and matches its operational definition.
Reliability is necessary for validity, but it is not sufficient by itself. A measurement can produce consistent results while still measuring the wrong concept. For example, a bathroom scale may consistently display inaccurate weights, while an aggression assessment may be valid if observers agree when evaluating aggressive behavior.
Reliability consists of stability reliability and equivalence reliability. Stability reliability is commonly assessed through test-retest procedures, using the same measurement at different times, and is appropriate only when the phenomenon remains relatively unchanged. When ratings are made by an observer, this consistency is called intraobserver or intrarater reliability.
Equivalence reliability includes inter-item consistency, parallel forms, and interobserver agreement. Inter-item consistency evaluates the relationship among questions designed to measure the same concept and is commonly assessed using Cronbach’s alpha. Parallel-forms reliability compares two equivalent measurements, while interobserver reliability evaluates the agreement between different observers.
Several factors can affect reliability, including test length, the interval between test and retest, score variability, guessing, and changes in the testing situation. Questions about past events may also be less reliable when they are overly specific and exceed the respondent’s memory capacity. Therefore, measurement procedures should be designed consistently and appropriately for the phenomenon being studied.
Validity is closely related to the purpose for which an assessment is used. The appropriate question is not whether an instrument is absolutely valid, but whether the conclusions based on it are valid for a particular purpose. Major types of validity include measurement validity, internal validity, external validity, and statistical conclusion validity.
Face validity examines whether a measurement appears appropriate for the concept being assessed, while content validity ensures that all dimensions of a concept are covered. Empirical or criterion validity compares the results of a measurement with another accepted measure, either at the same time through concurrent validity or by predicting future outcomes through predictive validity. In diagnostic testing, sensitivity refers to the ability to identify cases, specificity refers to the ability to identify noncases, and positive and negative predictive values explain the meaning of positive and negative test results.
Construct validity is used when no clear criterion or gold standard is available. It is assessed by examining whether a measurement relates to other measures as predicted by theory, whether its items are internally consistent, and whether it is unrelated to irrelevant concepts. Convergent validity is achieved when a measure correlates with other measures of the same concept, while discriminant validity is achieved when it does not strongly correlate with measures of different concepts. Statistical validity, or statistical conclusion validity, evaluates whether the conclusions about relationships between variables are accurate.