---
title: "Several Annotators per Image"
description: "Several annotators per image in LumiScan Annotation Web: blind annotation, automatic consensus, review of differences and agreement statistics in detail."
product: "LumiScan Annotation Web"
component: "lumiscan-annotation-web"
version: "2.2.0"
display_version: "2.2.0.0"
latest_version: false
language: "en"
url: "https://docs.hdvisionsystems.com/en/lumiscan-annotation-web/2.2.0/several-annotators-per-image.html"
---

# Several Annotators per Image – LumiScan Annotation Web

With **Annotators per image** set to 2 or more in the [project settings](https://docs.hdvisionsystems.com/en/lumiscan-annotation-web/2.2.0/project-settings.html), every image is annotated by that many people independently. The server compares their boxes. A step-by-step guide is in [Check Annotations with Two Annotators](https://docs.hdvisionsystems.com/en/lumiscan-annotation-web/2.2.0/how-tos/check-annotations-with-two-annotators.html).

[Video](https://docs.hdvisionsystems.com/en/lumiscan-annotation-web/2.2.0/_images/videos/two-annotators-and-review.mp4)

## Modes

Annotators see the mode **Annotate**, reviewers the mode **Review** and the **Statistics**. People with both roles switch at the top.

## Annotate (blind)

- Everyone draws their own version without seeing the others' boxes.
- The versions are stored apart from the reference, in `<folder>/.annotations/<image>/<user>.tsv` (same format as the CSV; training ignores them).
- Several people can annotate the same image at the same time.
- The image list shows **annotated / required** (`1/2`), and whether you have done it. Annotators see how many have annotated, not who.
- **Only images I have not annotated yet** filters the list.

## Automatic consensus

When the last required annotator saves, the server compares all versions. They **agree** when

- every object was drawn by everyone (nobody missed a box),
- matching boxes overlap at least the set value (**Agree: overlap ≥ %**, default 85 %, measured as rotated intersection over union),
- their angles differ less than the set limit (default 5°), with the bottom edge on the same side,
- they have the same label.

Then the server writes the **mean boxes** as the reference annotation `<image>.csv`, the file training reads, with `auto-consensus` as the user in the `.json` history. Otherwise the image goes to review.

Boxes of different annotators are matched with an optimal assignment (Hungarian method) on the rotated overlap.

## Review

![Review mode with the boxes of two annotators in their colours, red numbered markers at the differences and the differences panel on the right](https://docs.hdvisionsystems.com/en/lumiscan-annotation-web/2.2.0/_images/screens/review.jpg)

- The image list shows the state of each image: open, **to check**, **accepted** (agreed automatically), **done** (reviewed). It is sorted by the largest differences first; **Sort** changes that.
- Every annotator has a colour; the legend switches individual annotators on and off.
- Red numbered markers show the differences: box missing for someone, overlap too small, angle, bottom edge on the other side, different label.
- For each difference: **Show** zooms to it; the buttons take one person’s box, the **Average** or **None** into the reference.
- The boxes in the editor can also be changed freely before saving.
- **Save** writes the reference annotation. `.annotations/<image>/consensus.json` records who decided. A reviewed reference is never overwritten automatically, also if an annotator saves again.

## Statistics

**Statistics** shows the agreement for one folder or the whole project:

![Agreement between annotators: summary figures, per annotator, pairs of annotators, differences by kind, largest differences](https://docs.hdvisionsystems.com/en/lumiscan-annotation-web/2.2.0/_images/screens/statistics.png)

|   |   |
| --- | --- |
| Summary | Images with enough annotators, share of complete images that fully agree, mean overlap, share of objects that agree; images open, to check, agreed automatically, reviewed |
| Per annotator | Overlap with the consensus; systematic deviation of width, height and angle from the mean box; boxes only they drew, boxes they missed; other labels |
| Pairs of annotators | Agreement (F1) and mean overlap per pair |
| Differences by kind | How often each kind of difference occurs |
| Largest differences | The images with the most and largest differences; click to review |