Finding Objects on an Image Stream (Experimental)

The image stream is experimental. Its nodes and settings may still change. Use it for evaluation, and verify it with your cameras and your Data Layer before relying on it.

find-object searches one image per call: your program sets input/input-image-address and writes to execute. An image stream works the other way round. The Cam Driver publishes every image it takes on <camera name>/control/publish/output/stream/frame, with its size, its pixel format and its lineage (see the Cam Driver manual, Streaming Images). A module subscribes to the stream of one camera, searches each frame as it arrives, and publishes one result per frame on control/subscribe/output/stream/result.

Each frame is searched with the inputs of find-object as they are at that moment: the object, the confidence threshold, the crop region and every other node under find-object/input/. The one input the stream does not use is input/input-image-address: the frame takes its place. If you change an input while the stream runs, the next frame is searched with the new value.

Each result carries the lineage of its frame - which camera took it, in which run of the camera’s stream, as which frame of the run - with the module’s own step added. See The lineage of a result.

By default the module never falls behind its camera. When frames arrive faster than the module searches them, it takes the newest frame each time the previous search is finished. The frames in between are counted as skipped. How many frames may wait is set with the queue size, see The queue size.

A module takes the stream of one camera. For another camera, or other settings, use another module.

The stream nodes, and how a stream is started and stopped, are the same in every LumiScan app that consumes a stream; Streams (Experimental) in the Data Layer manual describes them in general. This page describes what is particular to the Object Finder.

The stream’s nodes

Every module has the stream’s nodes, whether it subscribes to a stream or not. No restart is needed to start using them. All addresses below start with hdvisionsystems/lumiscan-object-finder/<module-id>/. They sit beside find-object/, not inside it.

Table 1. Methods
Address Meaning

control/subscribe/execute

Checks the five inputs of control/subscribe together, saves them in the module’s settings file, then applies them: a running stream follows on its next cycle, with no restart. Writing an input alone changes nothing. See Changing the source.

control/unsubscribe/execute

Drops the source named by control/unsubscribe/input/address, which ends stream mode too. The tag stays. Saved in the module’s settings file like a subscribe.
Table 2. Inputs of control/subscribe
Address Type Meaning

control/subscribe/input/enable

bool

Stream mode: true runs the stream, false stops it.

control/subscribe/input/address

string

The one source: a camera’s stream folder, for example hdvisionsystems/lumiscan-cam-driver/camera1/control/publish/output/stream. The module reads its frame node. Another address replaces the source.

control/subscribe/input/tag

string

The tag the module puts on its results: the module’s step in the lineage of each result carries it, so whoever reads a result downstream sees which module made it. Empty: the module id, the last segment of the module’s address, for example module-1. A new tag marks the next result; the source stays subscribed.

control/subscribe/input/queue-size

uint32

How many frames may wait while a frame is searched: 1 to 64. See The queue size.

control/subscribe/input/keep-latest-output

bool

true to write each stream result to find-object/output/result as well. See Keeping the latest output.

The inputs start with what is applied at the start of the application, so a client can change one of them and execute.

What the source must hold: read hdvisionsystems/lumiscan-object-finder/system/meta/subscription-schema, the same for every module. It is a JSON Schema of the frame node of the source, a stream_image with its lineage and its image, in its JSON form. See Data Layer Reference.

Table 3. Input of control/unsubscribe
Address Type Meaning

control/unsubscribe/input/address

string

The source to drop: the address setting/stream/address shows. A / at either end does not count.
Table 4. Settings
Address Type Meaning

setting/stream/enable, setting/stream/address, setting/stream/tag, setting/stream/queue-size, setting/stream/keep-latest-output

as the inputs

What control/subscribe/execute or control/unsubscribe/execute applied last, as the settings file keeps it. Read-only.

setting/settings-hash

string

The hash of what the results are made with, as 16 hex digits. Read-only. See The settings hash.
Table 5. Outputs
Address Type Meaning

control/subscribe/output/stream/active

bool

true while the stream runs.

control/subscribe/output/stream/result

flatbuffer stream_object_matches

The matches found in each frame, and the frame’s lineage. Subscribe to it to receive each one. See The result of a frame.

control/subscribe/output/stream/last-failure

string

Why the last frame that could not be searched failed. It names the source by its address, see The result of a frame. Emptied when the stream starts.

control/subscribe/output/stream/statistics/…​

uint64

Counters of the current or the last stream, see Reading the statistics.

The stream publishes its results on control/subscribe/output/stream/result. Of the nodes of find-object it writes only find-object/output/result, and only with keep-latest-output on: status-code and status-text keep what the last call of the method left there.

Setting it up

  1. Make sure the Cam Driver’s stream runs: its hdvisionsystems/lumiscan-cam-driver/<camera name>/control/publish/output/stream/active is true.

  2. Set the inputs of find-object as for a single search, see Finding an Object. A call of find-object/execute on the camera’s output/image is a quick way to check them.

  3. Write the camera’s stream folder, for example hdvisionsystems/lumiscan-cam-driver/camera1/control/publish/output/stream, to control/subscribe/input/address. Leave control/subscribe/input/tag empty to mark the results with the module id, or write a tag of your own to it.

  4. Subscribe to control/subscribe/output/stream/result.

  5. Set control/subscribe/input/enable to true, then write to control/subscribe/execute. Wait until control/subscribe/output/stream/active is true. The results arrive from then on.

  6. To stop, set control/subscribe/input/enable to false and write to control/subscribe/execute again. The frame being searched is finished first.

On ctrlX OS, use the Data Layer view of the ctrlX OS web interface or your PLC program for these steps.

On a PC, the same steps with krxclient, the command line client of LumiScan Keryx that stands in for a PLC. Add the --keryx-backend and --keryx-url options your broker needs (see krxclient):

krxclient write hdvisionsystems/lumiscan-object-finder/module-1/control/subscribe/input/address hdvisionsystems/lumiscan-cam-driver/camera1/control/publish/output/stream
krxclient write hdvisionsystems/lumiscan-object-finder/module-1/control/subscribe/input/enable true
krxclient execute hdvisionsystems/lumiscan-object-finder/module-1/control/subscribe/execute
krxclient subscribe hdvisionsystems/lumiscan-object-finder/module-1/control/subscribe/output/stream/result --count 5

krxclient subscribe shows a flatbuffer by its size only. It tells you that results arrive, not what they hold.

Good to know:

  • Stream mode is a level, not a command, and the stream can be started before the Cam Driver’s stream: results arrive once frames do. See Starting and stopping a stream.

  • What control/subscribe/execute applies is saved in the module’s settings file. After a restart of the application the module starts as it was: one that streamed streams again at once, without your program executing anything. See Kept in the settings file.

Changing the source

A module takes one source. A PLC or another client changes it while the application runs, with or without a running stream:

  1. Write the new values to the inputs of control/subscribe - only those you change: the others hold what is applied.

  2. Write to control/subscribe/execute. The inputs are checked together, saved in the module’s settings file, and only then applied: setting/stream/ shows them at once, and a running stream follows on its next cycle. Another address replaces the source; the frames the old one had waiting are counted as skipped. A new tag, queue size or keep-latest-output needs no new subscription: a new tag marks the next result.

  3. To drop the source, write its address to control/unsubscribe/input/address, then write to control/unsubscribe/execute. Stream mode ends with it. The inputs of control/subscribe are set to what it applied: stream mode off, no address. The tag stays: it marks the module’s results, not the source.

When the settings file cannot be written, nothing changes. See Kept in the settings file.

The methods answer with a result whose name depends on the Data Layer (see Results of a method call):

ctrlX Data Layer LumiScan Keryx (PC) When

DL_OK

Ok

Done.

DL_INVALID_VALUE

Rejected

control/subscribe/execute: stream mode without an address, or a queue size outside 1 to 64. control/unsubscribe/execute: the address is not the source, or there is no source.

DL_CREATION_FAILED

CreationFailed

The settings file could not be written, for example because it is not valid JSON. Nothing changed.

These methods have no status-text. The reason for a refusal is written to the application log.

When the source cannot be subscribed to, the module tries it again every 5 seconds while the stream runs, and logs the first failure.

The result of a frame

control/subscribe/output/stream/result holds a FlatBuffers table of type stream_object_matches (file identifier HSOM), registered on the Data Layer as types/hdvisionsystems/lumiscan-object-finder/stream_object_matches. Its compiled schema is installed as bfbs/stream_object_matches.bfbs.

Field Meaning

lineage

Where the result came from, see The lineage of a result.

result

The matches, as a hdv.object_find.Result: the same table find-object writes to output/result for the same picture, described in Finding an Object. image_width and image_height are the frame’s size.

A read of control/subscribe/output/stream/result returns the latest result; before the first frame, an empty one. The last result stays after the stream stops.

A frame is decoded as find-object decodes the image node, so the same pixel formats are accepted, see Requirements. A frame and the same picture read through input/input-image-address give the same matches.

A frame that cannot be searched gives no result. It is counted in statistics/failed-frames, and control/subscribe/output/stream/last-failure says why, for example:

a frame of 'hdvisionsystems/lumiscan-cam-driver/camera1/control/publish/output/stream' failed: invalid input: object-id is empty - name the object to search for, see system/objects

The reasons are the ones find-object gives in its status-text: an input it would refuse, an object that is not listed on system/objects or has no picture, a crop region that does not fit the frame, a region the detector cannot search.

The lineage of a result

lineage.steps lists where the result came from, origin first:

  1. The camera’s step, as the Cam Driver made it: producer is the camera’s address, hdvisionsystems/lumiscan-cam-driver/<camera name>; run when the camera’s stream started; sequence the frame’s number in that run; external_id the camera’s own frame number; settings_hash the camera’s settings; tag the Cam Driver’s tag for the camera, by default its module id, for example camera1.

  2. The module’s step: producer is the module’s address, hdvisionsystems/lumiscan-object-finder/<module-id>; run when the module’s stream started (each start is a new run); sequence the result’s number in the run, from 1; settings_hash the module’s settings, see The settings hash; elapsed_us the microseconds from the frame’s arrival at the module to its result, waiting included; time_us when the result was made, in microseconds since the epoch; tag the module’s tag, control/subscribe/input/tag, or the module id when it is empty.

So a result names its frame exactly: the camera by producer, the frame by run and sequence. To know which camera a result came from, read the producer of its first step. A gap in the camera step’s sequence between two results of the same camera is frames that were skipped, failed or lost.

A lineage holds at most 16 steps. A longer one keeps the origin and the newest steps, and dropped counts the steps left out.

A frame that carries no lineage gives a result with the module’s step alone. A value on a stream’s frame node that is not a stream_image is ignored, and logged once.

The settings hash

setting/settings-hash, and the settings_hash of the module’s step in each result, identify what the results are made with. The node shows the hash as 16 hex digits; the step holds the same value as a number. The hash covers:

  • every input of find-object the search reads, exactly as it was read for the frame - all of them but input/input-image-address;

  • the object’s picture, by its content: a new picture saved under the old object id is a new hash;

  • the application’s name and version.

It does not cover stream mode, the source, the queue size or keep-latest-output: they choose and pace the frames, they do not change what is found in them.

The stream compares the picture’s content with the picture the module’s detector was trained on, for every frame. When they differ, the detector is trained again before the frame is searched, so a result is never made with a picture its hash does not name. find-object on its own does not compare the content: it trains again when a detector setting changes or the object list is refreshed, see Object Storage.

setting/settings-hash is updated about every half second, also while the stream is off. The first time a hash appears, the module writes the settings it stands for to settings-history/<module-id>-<hash>.json in the application’s configuration directory, so a hash found in a result can be looked up later. The application does not delete these files. A file looks like this, shortened:

{
    "format": 1,
    "settings": {
        "find-object/input/confidence-threshold": 0.6,
        "find-object/input/object-id": "screw-m6",
        "find-object/template": {
            "file": "<object directory>/screw-m6.png",
            "fnv1a64": "<16 hex digits>"
        },
        "software.name": "lumiscan-object-finder",
        "software.version": "<version>"
    },
    "settingsHash": "<16 hex digits>"
}
A result carries the hash only. A different hash says that something the results are made with changed, not what changed. To find out, compare the two files in settings-history that the hashes name, by hand. Every input the search reads is hashed, so a change that does not alter a result can still change the hash.

Reading the statistics

The counters start from zero each time the stream starts, are updated about four times a second while it runs, and keep their last values after it stops. All addresses start with control/subscribe/output/stream/statistics/.

Counter Meaning

elapsed-ms

Milliseconds since the stream started. Divide a counter by it for a rate.

received-frames

Frames the subscription delivered.

processed-frames

Frames searched, each with a result published on control/subscribe/output/stream/result.

skipped-frames

Frames never searched: pushed out of the queue by newer ones while a frame was searched (see The queue size), or waiting when the source was replaced or dropped.

failed-frames

Frames that could not be searched. control/subscribe/output/stream/last-failure says why the last one failed, and the log names the failures, at most one line every 5 seconds.

lost-frames

Frames dropped before they arrived, counted from gaps in the camera’s sequence within one run of its stream. A new source, or a new run of the camera’s stream, starts the count again.

processing-us

Microseconds all searches took, summed, without the time a frame waited for find-object. Divide by processed-frames for the mean.

Frames still waiting when the stream stops are dropped without being counted.

Kept in the settings file

Stream mode and the source are settings of the module. They are kept in the stream member of the module’s settings file, <module-id>.json in the application’s configuration directory - the file system/save-module-inputs writes the inputs to.

control/subscribe/execute and control/unsubscribe/execute save what they apply there themselves, all five values in one write, before they apply them. When the file cannot be written, the method answers CreationFailed (ctrlX: DL_CREATION_FAILED) and nothing changes. After a restart the module subscribes to the same source and, when stream.enable is true, starts its stream at once: nobody needs to execute anything.

Saving changes only the stream member; the rest of the file stays as it was. The module makes the file when there is none yet - a module whose inputs were never saved has none. system/save-module-inputs in turn keeps the stream member when it rewrites the file; the inputs of control/subscribe are not saved as the module’s inputs.

Platform Configuration directory

ctrlX OS

/var/snap/rexroth-solutions/common/solutions/activeConfiguration/lumiscan-object-finder

PC (Windows)

%APP_RESOURCE_DIR%\etc\lumiscan-object-finder; without APP_RESOURCE_DIR, C:\ProgramData\HD Vision Systems GmbH\LumiScan Vision Suite\lumiscan-obj-find\etc\lumiscan-object-finder

The stream member, beside the inputs of find-object (shortened here):

{
    "find-object": {
        "object-id": "screw-m6"
    },
    "stream": {
        "enable": true,
        "address": "hdvisionsystems/lumiscan-cam-driver/camera1/control/publish/output/stream",
        "queueSize": 1,
        "keepLatestOutput": false
    }
}
Key Required Meaning

enable

No

Stream mode: true starts the stream at the start of the application. Needs an address. false if left out.

address

No

The one source: a camera’s stream folder.

tag

No

What marks the module’s results: the tag of the module’s step in their lineage. The module id if left out.

queueSize

No

How many frames may wait: 1 to 64, 1 if left out.

keepLatestOutput

No

true to write each stream result to find-object/output/result as well. false if left out.

onEntry

No

The steps applied to each frame as it arrives, each an object with a kind, as every LumiScan application’s stream has them. This application has none yet: leave it out, or empty ([]). A step starts the module out of stream mode, with a warning in the log; saving keeps it.

control/subscribe/execute and control/unsubscribe/execute write all five.

Good to know:

  • The file is read once, when the module is restored at the start of the application: a module in the saved module list, or module-1 on a fresh installation. Changes made to the file by hand take effect at the next start.

  • A module made with system/create-module starts with the stream off, no source, a queue size of 1 and keep-latest-output off, whatever a file of its name holds. What it applies from then on is saved to that file.

  • To have a module and its stream back after a restart, save the module list too, with system/save-current-modules. Saving a list that no longer names a module deletes that module’s file, and its stream settings with it. See Module Management.

  • A value that cannot be used is ignored, and logged: an enable or keepLatestOutput that is not true or false, an address or tag that is not a string, a queueSize outside 1 to 64 (1 is used). Stream mode without an address is ignored: the module starts out of stream mode.

  • The subscriptions member of earlier versions is no longer read, and the module logs that it moved: give the source as address.

Do not add comments to <module-id>.json. The stream reads a file with comments, but the module’s inputs are then not restored at start: the log reports the file as unreadable.

The queue size

control/subscribe/input/queue-size sets how many frames may wait while a frame is searched: 1 to 64, 1 by default. It is saved as stream.queueSize, and setting/stream/queue-size shows the value in use. A frame that arrives when that many wait pushes out the oldest of them, which is counted in statistics/skipped-frames. A new queue size applies from the next frame that arrives, with no new subscription.

Queue size Effect

1 (default)

The module searches the newest frame. It never falls behind; when the search is slower than the camera, it skips frames.

More than 1

Short bursts of frames are all searched, and fewer are skipped. When the search is slower than the camera for longer, the module falls behind: it searches frames up to that many searches old. Each waiting frame holds a whole image in memory.

Keeping the latest output

With control/subscribe/input/keep-latest-output true - applied by control/subscribe/execute, saved as stream.keepLatestOutput - the module also writes each stream result to find-object/output/result, as find-object/execute writes it, once the result is published on control/subscribe/output/stream/result. A client that reads find-object/output/result then has the newest frame’s result without subscribing to the stream. status-code and status-text are not written: they keep what the last call of find-object left there. It costs a copy of each result. Off by default.

find-object while the stream runs

The stream and find-object use the same inputs and the module’s one trained detector, and they never run at the same time on one module:

  • A frame that arrives while find-object runs waits until the method has finished. It is delayed, not failed.

  • A call of find-object/execute while a frame is searched answers busy: output/status-code is -1, the write answers TooManyOperations (ctrlX: DL_TOO_MANY_OPERATIONS), and nothing is done. output/result keeps its value, as for any busy call (see Execute a Method). With a camera streaming, this happens often. Stop the stream before you call the method, or call it again.

  • system/remove-module also answers busy (-1) while a frame of that module is searched. Stop the stream first. Removing a module stops its stream.

  • system/create-object has a lock of its own and is not held up by the stream.

Other modules are not affected: each module has its own lock and its own stream.

Limitations

  • The stream has been tested on Windows with LumiScan Keryx and simulated images only. It has not been verified on Linux - on ctrlX OS or Ubuntu - on the ctrlX Data Layer, or with a real camera.

  • LumiScan Cockpit does not draw the stream’s results yet.

  • Changes made to the settings file by hand take effect at the next start of the application.

  • The hash covers every input the search reads. It changes with some edits that do not change a result.