INTER-RATER RELIABILITY
Inter-Rater Reliability Analysis, as a method of measuring the agreement among coders in their analysis of unstructured data, is more closely aligned with positivistic research designs than with qualitative methodologies, although there is some justification for use of this method with large projects using multiple researchers. I include these references to provide visitors with a clearer understanding of the purpose and methods of inter-rater reliability analysis.
Bourdon, S. (2001). From Measuring Rigour To Fostering It : Automating Intercoder Reliability Verification. QSR Nsight, (16), 11.
Carey, J. W., Morgan, M., and Oxtoby, M. J. Intercoder Agreement in Analysis of Responses to Open-Ended Interview Questions: Examples from Tuberculosis Research
Intercoder Agreement in Analysis of Responses to Open-Ended Interview Questions (10/96 - (PDF)
The authors here argue that the use of intercoder agreement is necessary to guard against the introduction of subjective bias in the coding and analysis of qualitative data, which otherwise could have "adverse consequences" and lead to "inappropriate theoretical conclusions" and/or "ineffective or harmful recommendations and interventions." The authors then make something of a leap by suggesting this approach should apply to all qualitative research; the implication being that intercoder agreement will ultimately lead to more credible findings.
Intercoder agreement, as I understand it, is designed to provide a high degree of quality assurance as to the accuracy of a given interpretation. The problem is this: intercoder agreement is arrived at through an interpretive process (albeit through the lens of multiple researchers). It is here I believe the authors confound the notions of interpretive accuracy and interpretive authenticity by suggesting that a process that addresses interpretive accuracy will necessarily result in an authentic interpretation of the data and thus produce more credible findings. It is my contention that interpretive authenticity is best achieved when researchers subject their interpretations to the scrutiny of the study participants. It is they who know best the meanings they intended to convey through their responses.Morse, J. M. (1997). Perfectly healthy, but dead; the myth of inter-rater reliability. Qualitative Health Research, 7(4), 445-447.
Trochim, B. What is Reliability? Discusses reliability from a totally quantitative perspective. Includes a section on Inter-Rater or Inter-Observer Reliability.
 |