Table 9 Comparison of the inter-annotator and self-agreement over four evaluation measures
From: Stance and influence of Twitter users regarding the Brexit referendum
Inter-annotator agreement | Annotators’ self-agreement | |
|---|---|---|
No. of overlapping examples | 6807 | 2316 |
\({Accuracy}\) (\(\,-,0,+\)) | \( 77.2 \% \) | \( 86.4 \% \) |
\(\overline{F_{1}}(-,+)\) | \( 74.6 \% \) | \( 87.1 \% \) |
\({Accuracy}\,\!\pm \!\,1\) ( \(-,+\)) | \( 96.0 \% \) | \( 97.4 \% \) |
Alpha | \( 67.7 \) (4604 examples) | 82.6 (1969 examples) |