1 team functionality with multiple user accounts
2 easy to use workflow for double annotation where each text is annotated by exactly two annotators. The software should make sure that a text is never shown to more than 2 annotators and never shown to the same annotator twice
3 make it easy to review the 2 versions and solve conflicts
4 smarter alternative to review would be a warning system that identifies annotations that may have errors (because a model trained on the other data predicts a different result) and automatically flags it for review by another annotator
5 stats on the annotators: speed, accuracy, statistics on how frequently they assign different labels to detect potential misunderstandings of the annotation schema
6 GUI with overview of all annotation datasets, with stats like % finished annotating (with stages for double annotation and review), the types of annotation done, frequencies of labels to detect imbalances
7 functions to mass-edit the annotations, like renaming or removing an entity type