Finding an Object

<module>/find-object searches one image, once, for a known object.

Inputs

Node Default Meaning

input/input-image-address

(empty)

Address of the image node to search.

input/object-id

(empty)

Which object to look for. Required - an empty value is refused.

input/confidence-threshold

0.6

Smallest acceptable score, 0 to 1.

input/nms-min-dist

0.0

Minimum distance in pixels between two reported matches.

input/pyramid

[1,2]

Image pyramid levels the search runs at. Not a speed setting - see Speed.

input/result-coordinate

IMAGE

IMAGE or CROP - see below.

input/crop/*

all 0

Region to search; all zero searches the whole image.

input/preprocess/bin-size

1

Search at a lower resolution: faster, less precise. See Bin size.

input/preprocess/blur-*

off

Optional blur - see Tuning the Search.

Running it

Set the inputs, then write to <module>/find-object/execute:

module-1/find-object/input/input-image-address = hdvisionsystems/lumiscan-cam-driver/camera1/output/image
module-1/find-object/input/object-id           = screw-m6
module-1/find-object/input/confidence-threshold = 0.7
module-1/find-object/input/nms-min-dist         = 40
write module-1/find-object/execute

Then read output/status-code and output/result.

Set nms-min-dist to roughly half the size of the part in pixels. Without it a single part is often reported several times over, as a cluster of slightly shifted matches.

One object per search

input/object-id names one of the entries on system/objects and is required: an empty value is refused with status code -2.

There is no "search for everything" run. A detector is built around one picture, so searching several objects means several detectors, and each one re-processes the whole image - which is the expensive part. Running find-object once per object makes that cost visible rather than hiding it behind an empty input.

To sort mixed parts, run the method once per object and compare the scores.

The result

output/result is an object_find_result flatbuffer.

Field Meaning

matches[].x, .y

Top left corner of the matched template, in pixels.

matches[].width, .height

Size of the template that matched, so the match can be drawn.

matches[].score

0.0 to 1.0 similarity.

matches[].object_id

The object that was found; the same on every match of one run.

best_index, best_score, best_object_id

The highest scoring match.

image_width, image_height

The frame the coordinates are in.

confidence_threshold

The threshold that was applied.

timestamp

Microseconds since epoch, stamped when the run finished.

Matches are sorted by score, so matches[0] is the best one.

There is no rotation in the result: the reported rectangle is axis aligned. If you need to know which orientation was found, give each orientation its own object - screw-m6-front, screw-m6-tilted - and search for them in turn. The answer is then the object_id that matched.

Image and crop coordinates

input/result-coordinate chooses the frame positions are measured in:

  • IMAGE (default) - the full image at input-image-address. This is what a robot pick needs.

  • CROP - the searched region, whose top left corner becomes (0,0).

image_width and image_height follow the same choice, so normalising a coordinate by them is correct either way.

The region is searched exactly as given

Any size works, and no part of input/crop is dropped. cv::linemod does need the image it is handed to be a particular multiple, but the detector pads up to it internally rather than demanding a size it likes, so nothing you configure has to satisfy a rule.

This used to trim instead, costing up to 7 pixels at the right and bottom edges - 31 at bin-size 4. Trimming was worse than it looked: it threw away columns the camera really captured, and left cv::linemod’s own dead border - it ignores the outer ring of whatever it is given - sitting on the pixels that survived. Measured, an object at the last usable position scored 0.79 trimmed against 1.00 padded.

The padding replicates the edge pixels outward, so it introduces no gradients of its own and cannot be mistaken for a shape, and a match is never reported from inside it. It is added at the right and bottom only, so every reported position is still a position in the image you configured.

What an empty result means

An empty result with status-code 1 means the object was not in the image, and nothing else.

Every other reason a search could come back empty - the object has no picture, the image is unusable, a pyramid level is invalid, bin-size left too little of the object to recognise - is detected before the search runs and reported as a negative status code. See Execution Status.

Detection speed

The picture is compiled into a detector the first time it is used, and that detector is reused for as long as the settings and the object stay unchanged. Changing any input, or saving a new template, rebuilds it on the next run.

Compiling is by far the most expensive part of a run, so a production loop that executes repeatedly with unchanged settings pays that cost once. Changing a setting before every call pays it every time.

To make the search itself faster, narrow input/crop first and then raise input/preprocess/bin-size. Raising input/pyramid does not help - see Speed.

Searching every frame of a camera

find-object searches one image per call. To have a module search every frame a camera streams, with the same inputs, see Finding Objects on an Image Stream (Experimental).