Skip to main content

Table 9 Comparison of the inter-annotator and self-agreement over four evaluation measures

From: Stance and influence of Twitter users regarding the Brexit referendum

  Inter-annotator agreement Annotators’ self-agreement
No. of overlapping examples 6807 2316
\({Accuracy}\) (\(\,-,0,+\)) \( 77.2 \% \) \( 86.4 \% \)
\(\overline{F_{1}}(-,+)\) \( 74.6 \% \) \( 87.1 \% \)
\({Accuracy}\,\!\pm \!\,1\) ( \(-,+\)) \( 96.0 \% \) \( 97.4 \% \)
Alpha \( 67.7 \) (4604 examples) 82.6 (1969 examples)