Introduction

Inter-rater reliability is a measure that indicates the level of agreement between two or more coders on a project. This measure is used to ensure that the study analysis is conducted with rigor and that the data collected and analyzed is consistent among all researchers. High inter-rater reliability scores are significant because the relationship between coders is crucial to the validity of the research project. The three main ways to measure inter-rater reliability in qualitative coding include using Cohen’s Kappa score, Intraclass Correlation Coefficient, and Percentage Agreement scores.

Methods to Measure Inter-rater Reliability (IRR)

There are multiple ways to measure inter-rater reliability in qualitative coding.

1. Cohen’s Kappa – Cohen’s Kappa is a statistical measure that is used to measure the agreement of coding between multiple coders. This measure is best used with categorical data and ranges from 1 to -1 with 1 indicating perfect agreement and -1 indicating complete disagreement. Zero suggests there is no better agreement than chance. This a more nuanced understanding of the reliability and congruence of the raters.[1] 

2. Intraclass Correlation Coefficient – ICC is used to as a method to measure reliability of measurements made by multiple raters. It is most useful when the data is continuous rather than categorical. The ICC values range between 0 and 1, with values closest to 1 indicating higher reliability. This is different from Cohen’s Kappa because it is ideal for continuous data while Cohen’s Kappa is better for categorical data, and ICC can be used with more than two raters unlike Cohen’s Kappa.

3. Percentage Agreement – Percentage agreement is the simplest way to measure inter-rater reliability by calculating the number of times that the coders agree without the possibility of chance agreement. Because this method does not account for agreements that might have occurred for chance, this is a less robust and rigorous method for measuring inter-rater reliability.[2]

Conclusion

High inter-rater reliability scores are significant because the relationship between coders is crucial to the validity of the research project. Cohen’s Kappa is a statistical measure that is used to measure the agreement of coding between multiple coders. ICC is used to as a method to measure reliability of measurements made by multiple raters. It is most useful when the data is continuous rather than categorical. Percentage agreement is the simplest way to measure inter-rater reliability by calculating the number of times that the coders agree without the possibility of chance agreement.

Take Away

This article explains how inter-rater reliability is used as a measure of reliability in qualitative coding. This is critical for ensuring validity in qualitative research coding.

[1] Nili, A., Tate, M., Barros, A., & Johnstone, D. (2020). An approach for selecting and using a method of inter-coder reliability in information management research. International Journal of Information Management, 54, 102154.s.

[2] O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: debates and practical guidelines. International journal of qualitative methods, 19, 1609406919899220.