Check Annotations with Two Annotators

Two people who annotate the same image independently rarely draw exactly the same boxes. Where they differ, the image is hard or the instructions are unclear. LumiScan Annotation Web uses this to improve the training data: the server compares the annotators, writes the reference itself when they agree and shows a reviewer exactly where they differ.

1. Switch it on

A project admin opens Settings and sets, under Several annotators per image:

  • Annotators per image: 2 (or more).

  • Agree: overlap ≥ %: how much two boxes must overlap to count as the same box, default 85 %.

  • Agree: angle within °: default 5°.

  • Write the reference automatically when all annotators agree: on.

2. Annotate blind

Annotators work as usual in the mode Annotate. Each sees only their own boxes, never the others'. The image list shows how many have annotated an image (1/2).

Annotate mode with two annotators per image: the image list shows 1/2 or 2/2 per image, the image shows only the annotator’s own boxes

When the last annotator saves:

  • all agree: same objects, same labels, overlap and angle within the limits. The server writes the mean boxes as the reference <image>.csv. The status says Saved. All 2 annotators agree: reference annotation written.

  • they differ: the image goes to review.

3. Review the differences

Reviewers switch to Review in the top bar. The list is sorted by the largest differences first.

Review mode: the boxes of the two annotators in blue and orange on the image, numbered red markers where they differ, and the list of differences on the right

Every annotator has a colour; red numbered markers show the differences:

  • a box only one person drew,

  • overlap too small,

  • angle too different,

  • bottom edge on the opposite side,

  • different label.

For each difference, the panel on the right offers the boxes of each person, Average or None. Click Show to zoom to it. When every difference is decided, Save writes the reference annotation. A reviewed reference is never overwritten automatically.

Differences panel: two annotators, two of seven objects differ; for each difference the buttons Show, the annotators' names, Average and None

4. Read the statistics

Statistics in the top bar shows the agreement for a folder or the whole project:

Agreement between annotators: images with two annotators, share of images and objects that agree, mean overlap; tables per annotator, per pair, differences by kind and the images with the largest differences

  • Per annotator: overlap with the consensus and systematic habits, for example boxes drawn wider than the others (width +3 %) or rotated.

  • Pairs of annotators: agreement (F1) and mean overlap.

  • Differences by kind and Largest differences: click an image to review it.

Use the per-annotator figures to improve the instructions: a systematic deviation usually means the rule was not clear.

More: Several Annotators per Image.