Table 1 Inter-annotator agreement for sentence recognition (SR), medical entity recognition (MER), and keyphrase extraction (KE) dataset
From: Sentences, entities, and keyphrases extraction from consumer health forums using multi-task learning
Phase | Agreement metrics | SR(%) | MER(%) | KE(%) |
|---|---|---|---|---|
Initial annotation | F1-score | 72.84 | 49.41 | 28.89 |
Cohen’s Kappa | 84.85 | 63.98 | 31.51 | |
Final annotation | F1-score | 88.61 | 64.83 | 35.01 |
Cohen’s Kappa | 94.63 | 76.51 | 44.85 |