How do you calculate the inter-rater reliability/correlation coefficient?

Question

I am doing a content analysis project for Psychology and have a question. Let's say that I have four raters who each want to record the number of instances of aggression in a TV show.

Rater 1: 10 instances
Rater 2: 5 instances
Rater 3: 7 instances
Rater 4: 9 instances

How can I calculate the correlation coefficient and the inter-rater reliability of the observers? I did a good amount of research on the subject, but am only a junior in high school, so statistical analysis is not my forte, per se. If anyone could explain how to do this, I'd be more than grateful.

Thanks!

Answer 1

You sure know how to hit a sore spot. :-)

The current "standard" for rater reliability is the Kappa coefficient, but there is much disagreement over it.

Here is a general article on the subject:
http://en.wikipedia.org/wiki/Inter-rater_reliability

For N raters, you want Fleiss' Kappa:
http://en.wikipedia.org/wiki/Fleiss%27_kappa

My feeling is that if you can explain this your teacher, you deserve an A. :-)

Answer 2

This Site Might Help You.

RE:
How do you calculate the inter-rater reliability/correlation coefficient?
I am doing a content analysis project for Psychology and have a question. Let's say that I have four raters who each want to record the number of instances of aggression in a TV show.

Rater 1: 10 instances
Rater 2: 5 instances
Rater 3: 7 instances
Rater 4: 9 instances

How can I...

Answer 3

Come on Tottenham you can beat Milan