Handbook of Inter-Rater Reliability

Low inter-rater reliability can jeopardize the integrity of scientific inquiries or have dramatic consequences in practice. In a clinical setting for example, a wrong drug or wrong dosage of the correct drug may be administered to patients at a hospital due to a poor diagnosis. Likewise, exam grades are considered reliable if they are determined only by the candidate's proficiency level in a particular skill, and not by the examiner's scoring method. The study of inter-rater reliability helps researchers address these issues using an approach that is methodologically sound. The 4th edition of this book covers Chance-corrected Agreement Coefficients (CAC) for the analysis of categorical ratings, as well as Intraclass Correlation Coefficients (ICC) for the analysis of quantitative ratings. The 5th edition however, is released in 2 volumes. The present volume 2, focuses on ICC methods whereas volume 1 is devoted to CAC methods. The decision to release 2 volumes was made at the request of numerous readers of the 4th edition who indicated that they are often interested in either CAC techniques or in ICC techniques, but rarely in both at a given point in time. Moreover, the large number of topics covered in this 5th edition could not be squeezed in a single book, without it becoming voluminous. Volume 2 of the Handbook of Inter-Rater Reliability 5th edition contains 2 new chapters not found in the previous editions, and updated versions of 7 chapters taken from the 4th edition. Here is a summary of the main changes from the 4th edition that you will find in this book:Chapter 2 is new to the 5th edition and covers various ways of setting up your rating dataset before analysis. Chapter 3 is introductory and an update of chapter 7 in the 4th edition. In addition to providing an overview of the book content similar to that of the 4th edition, this chapter introduces the new multivariate intraclass correlation not covered in previous editions. Chapter 4 covers intraclass correlation coefficients in one-factor models and has a separate section devoted to sample size calculations. Two approaches to sample size calculations are now offered: the statistical power approach and the confidence interval approach. Chapter 5 covers intraclass correlation coefficients under the random factorial design, which is based on a two-way Analysis of Variance model where the rater and subject factors are both random. Section 5.4 on sample size calculations has been expanded substantially. Researchers can now choose between the statistical power approach based on the Minimum Detectable Difference (MDD) and the confidence interval approach based on the target interval length. Chapter 6 covers intraclass correlation coefficients under the mixed factorial design, which is based on a two-way Analysis of Variance model where the rater factor is fixed and the subject factor random. The treatment of sample size calculations has been expanded substantially. Chapter 7 is new and covers Finn's coefficient of reliability as an alternative to the traditional intraclass correlations when they are not be applicable. Chapter 8 entitled "Measures of Association and Concordance" covers various association and concordance measures often used by researchers. It includes a discussion of Lin's concordance correlation coefficient and its statistical properties. Chapter 9 is new and covers 3 important topics: the benchmarking of ICC estimates, a graphical approach for exploring the influence of individual raters in low-agreement inter-rater reliability experiments, and the multivariate intraclass correlation. I wanted this book to be sufficiently detailed for practitioners to gain more insight into the topics, which would not be possible if the book was limited to a high-level coverage of technical concepts.