Inference on an Image Stream (Experimental)

Inference on an image stream is experimental (from v2.2.0). Its nodes may still change. Use it for evaluation, and verify it with your network and camera before relying on it.

The general concept of streams - producers and consumers, the result envelope and its lineage - is described in Streams. This page describes the stream of AI Object Matching. It works the same on the ctrlX Data Layer and on LumiScan Keryx.

On request, a module infers on one image per request: your program writes the image, or the Cam Driver captures one, and then asks for the inference. With an image stream, the Cam Driver publishes every image it takes on control/publish/output/stream/frame, with its size, its pixel format and its lineage (see the Cam Driver manual, Streaming Images). A module subscribes to the stream of one camera - its source - and infers on the images as they arrive, and publishes one result per image it infers. Each result carries the lineage of its image - which camera took it, in which run of its stream, as which image of the run - with the module’s own step added, see The lineage of a result.

By default the module never falls behind the camera. When images arrive faster than the network infers, it infers on the newest image each time the previous inference is finished. The images in between are counted as skipped. How many images may wait is set with the queue size, see The queue size.

A module has one source. To infer on the images of two cameras, use two modules.

On both Data Layers every image reaches the module, in order. Images dropped on the way are counted in statistics/lost-frames: each image carries its number in its camera’s stream.

Setting it up

Every module has the stream’s nodes below, whether it has a source yet or not: no restart is needed to start using them. Either

in the module configuration file, set the stream section, for example
```
"stream": {
    "enable": true,
    "address": "hdvisionsystems/lumiscan-cam-driver/camera1/control/publish/output/stream"
}
```
See Module Configuration File. A changed file applies at the next start or reset; or
at run time, write the inputs of control/subscribe and call it, see Subscribing at run time.

The image stream is inferred with the module’s network, class filters and remapping. For another network or other settings, use another module.

Kept in the settings

Stream mode and the source are settings of the module, in the stream section of its configuration file. Without that section, or without enable in it, the module is not in stream mode. Stream mode needs a source: a module configuration file with enable set to true and no address is not valid, and the module is not loaded. Neither is a module with a step in stream.onEntry: the section has the steps of every LumiScan application’s stream, and AI Object Matching has none yet.

control/subscribe/execute and control/unsubscribe/execute save enable, address, tag, queueSize and keepLatestOutput before they apply them. After a restart the module subscribes to the same source and, when stream.enable is true, starts its stream at once - nobody has to call the method again. Saving changes only those values: the rest of the file stays as it was, comments included.

Addresses

All addresses below start with hdvisionsystems/lumiscan-ai-object-matching/<network name>/. The stream follows the layout of an execute method: control/subscribe owns its inputs and outputs.

Table 1. Methods
Address	Meaning
`control/subscribe/execute`	Checks all of `control/subscribe/input/…` together, saves them in the module configuration file, then applies them: the stream follows on its next cycle, with no restart. See Subscribing at run time.
`control/unsubscribe/execute`	Drops the source named by `control/unsubscribe/input/address`, which ends stream mode too, and saves that. The tag stays.

Table 2. Inputs
Address	Type	At start	Meaning
`control/subscribe/input/enable`	bool	`stream.enable`, or `false`	`true` infers on the images of the source; `false` stops it.
`control/subscribe/input/address`	string	`stream.address`, or empty	The stream folder of the source, holding the camera’s `frame`, for example `hdvisionsystems/lumiscan-cam-driver/camera1/control/publish/output/stream`.
`control/subscribe/input/tag`	string	`stream.tag`, or empty	The tag the module puts on its results: the module’s step in the lineage of each result carries it, so whoever reads a result downstream sees which module made it. Empty: the module id, the last segment of the module’s address - `example` for `hdvisionsystems/lumiscan-ai-object-matching/example`. A new tag marks the next result; the source stays subscribed.
`control/subscribe/input/queue-size`	uint32	`stream.queueSize`, or 1	How many images may wait for the inference, 1 to 64. See The queue size.
`control/subscribe/input/keep-latest-output`	bool	`stream.keepLatestOutput`, or `false`	`true` also puts each stream result on the regular output. See Keeping the latest output.
`control/unsubscribe/input/address`	string	empty	The source `control/unsubscribe/execute` drops: the address on `setting/stream/address`.

Writing an input changes nothing by itself: only control/subscribe/execute applies the inputs. They start as what the module applies, and a reset sets them to the file’s values again, so a client can change one input and call the method.

What the source must hold: read hdvisionsystems/lumiscan-ai-object-matching/system/meta/subscription-schema (string, read only), the same for every module. It is a JSON Schema (draft-04) of the frame node of the source, a stream_image with its lineage and its image, in its JSON form. The node is experimental.

Table 3. Outputs
Address	Type	Meaning
`control/subscribe/output/stream/active`	bool	`true` while the module infers on its image stream.
`control/subscribe/output/stream/result`	flatbuffer `stream_detections`	The result of each image inferred, in the same format as `output/result`, and its lineage. Subscribe to it to receive each one. See The result of a streamed image.
`control/subscribe/output/stream/statistics/…`	uint64	Counters of the current or last stream, see Reading the statistics.

Table 4. Setting
Address	Type	Meaning
`setting/stream/enable`, `setting/stream/address`, `setting/stream/tag`, `setting/stream/queue-size`, `setting/stream/keep-latest-output`	as the inputs	What the module applies now, as saved in the `stream` section. A client finds the source’s images here: LumiScan Cockpit uses `setting/stream/address` to draw the results on the live images.
`setting/settings-hash`	string	The hash of the settings the results are produced with, as 16 hex digits. See The settings hash.

Unless keep-latest-output is set, the results of the stream are published only on control/subscribe/output/stream/result, and output/result, output/package-identifier, output/timestamp and output/image keep the result of the last inference on request.

The result of a streamed image

control/subscribe/output/stream/result holds a FlatBuffers table of type stream_detections, registered on the Data Layer as types/hdvisionsystems/lumiscan-ai-object-matching/stream_detections. Its compiled schema is installed as bfbs/stream_detections.bfbs. It has two fields: result, the same table as output/result (object_detection), and lineage, see The lineage of a result.

Three fields of result describe the image:

Field Meaning

Field	Meaning
`image_id`	The number of the image since the stream started: 1, 2, 3 and so on. A gap means the images in between were skipped or failed. A 64-bit number (`ulong`). In versions before October 2026 it was a 16-bit `short`, and wrapped to negative numbers after 32767 images.
`timestamp`	When the image arrived, in microseconds since the epoch.
`requested_timestamp`	When the inference on the image started, in microseconds since the epoch. The difference to `timestamp` is how long the image waited.

image_id

The number of the image since the stream started: 1, 2, 3 and so on. A gap means the images in between were skipped or failed. A 64-bit number (ulong). In versions before October 2026 it was a 16-bit short, and wrapped to negative numbers after 32767 images.

timestamp

When the image arrived, in microseconds since the epoch.

requested_timestamp

When the inference on the image started, in microseconds since the epoch. The difference to timestamp is how long the image waited.

The class filters in input/class-filters and the class index remapping of the module apply as they do on request. A class the network description does not name is published with its index.

Keeping the latest output

With keep-latest-output set, each stream result is also written to the module’s regular output, as an inference on request writes it: output/result, output/timestamp, and output/package-identifier, counted on from the inferences on request. A client that reads the regular output - a PLC program written for inference on request, for example - then sees the newest stream result there. The output image (output/image) is not drawn.

It costs a copy of every result, so it is off by default. The stream result on control/subscribe/output/stream/result is published first.

The lineage of a result

lineage.steps lists where the result came from, origin first:

The camera’s step, as the Cam Driver made it: producer is the camera’s address, hdvisionsystems/lumiscan-cam-driver/<camera name>, run when its stream started, sequence the image’s number in that run, external_id the camera’s own frame number, settings_hash the camera’s settings, and tag the Cam Driver’s tag for the camera, by default its module id, for example camera1.
The module’s step: producer is the module’s address, hdvisionsystems/lumiscan-ai-object-matching/<network name>; run when the module’s stream started (each start is a new run); sequence the result’s number in the run; settings_hash the module’s settings, see The settings hash; elapsed_us the microseconds from the image’s arrival to the result’s publication; time_us when it was published; tag the module’s tag, control/subscribe/input/tag, or the module id when it is empty.

So a result names its image exactly: the camera by producer, the image by run and sequence. A source changed while the stream runs keeps the module’s run: the results of the new camera are numbered on, and the producer of their first step names it. Two results of the same camera image, before and after a change of settings, differ in the settings_hash of the module’s step.

An image that carries no lineage - published by something other than the Cam Driver - gives a result with the module’s step alone. A value on the stream’s frame node that is not a stream_image is ignored, and logged once.

The settings hash

setting/settings-hash, and the settings_hash of the module’s step in each result, identify what the results are produced with. The hash changes when one of these changes:

the module configuration file’s settings, as the module was started with - except the stream section and externalImagePath, which choose the images rather than what is made of them;
the network description, and the network file it names, by its content - a new network under the old file name is a new hash; with simulateResult, that file’s content instead;
input/class-filters;
the application’s version.

A reset reads the network description and the network file again. The first time a hash appears, the module writes the settings it stands for to settings-history/<network name>-<hash>.json in the application’s configuration folder, so a hash found in a result can be looked up later.

A result carries the hash only. A different hash says that something the results are produced with changed, not what changed. To find out, compare the two files in settings-history that the hashes name, by hand. The same holds for the camera’s hash in the camera’s step; see the Cam Driver manual, Streaming Images, for what it follows.

Subscribing at run time

A PLC or another client changes the source and stream mode while the application runs, with or without a running stream:

Write the inputs that change to control/subscribe/input/… - for another camera, control/subscribe/input/address; for another mark on the results, control/subscribe/input/tag; to start or stop, control/subscribe/input/enable.
Write true to control/subscribe/execute.

The method checks all inputs together, saves them in the stream section, then applies them: setting/stream/… shows them at once, and a running stream follows on its next cycle. Another address replaces the source - no unsubscribe is needed first -, and the images the old camera had waiting are counted as skipped. The tag, the queue size and keep-latest-output apply without subscribing again: a new tag marks the next result.

To drop the source, write its address to control/unsubscribe/input/address and true to control/unsubscribe/execute. That ends stream mode too, and saves enable as false with no address. The tag stays: it marks the module’s results, not the source. control/subscribe/input/… are set to that as well, so a later control/subscribe/execute that changes another input does not bring the source back. A / at either end of an address does not matter.

The methods answer:

ctrlX Data Layer LumiScan Keryx When

ctrlX Data Layer	LumiScan Keryx	When
`DL_OK`	`Ok`	Done.
`DL_INVALID_VALUE`	`Rejected`	`control/subscribe`: `enable` is `true` and the address is empty, or the queue size is not 1 to 64. `control/unsubscribe`: the address is not the source, or there is no source. Nothing changed.
`DL_CREATION_FAILED`	`CreationFailed`	The module configuration file could not be written. Nothing changed: the stream stays as it was. The answer and the log say why.
`DL_RESOURCE_UNAVAILABLE`	`ResourceUnavailable`	The settings were edited on `system/settings` and the application waits for a restart.

DL_OK

Ok

Done.

DL_INVALID_VALUE

Rejected

control/subscribe: enable is true and the address is empty, or the queue size is not 1 to 64. control/unsubscribe: the address is not the source, or there is no source. Nothing changed.

DL_CREATION_FAILED

CreationFailed

The module configuration file could not be written. Nothing changed: the stream stays as it was. The answer and the log say why.

DL_RESOURCE_UNAVAILABLE

ResourceUnavailable

The settings were edited on system/settings and the application waits for a restart.

A reset (control/reset) reads the stream section of the module configuration file again and applies it, and sets the inputs to it: a running stream follows. So a file edited by hand applies with a reset.

When the subscription cannot be made, for example because the camera is not running yet, the module tries it again every 5 seconds while the stream runs, and logs the first failure.

The queue size

control/subscribe/input/queue-size, saved as stream.queueSize, sets how many images may wait for the inference: 1 to 64, 1 by default. An image that arrives when that many wait pushes out the oldest of them, which is counted in statistics/skipped-frames.

Queue size	Effect
1 (default)	The module infers on the newest image. It never falls behind; when the network is slower than the camera, it skips images.
More than 1	Short bursts of images are all inferred, and fewer are skipped. When the network is slower than the camera for longer, the module falls behind: it infers on images up to that many inferences old. Each waiting image holds a whole image in memory.

Queue size

Effect

1 (default)

The module infers on the newest image. It never falls behind; when the network is slower than the camera, it skips images.

More than 1

Short bursts of images are all inferred, and fewer are skipped. When the network is slower than the camera for longer, the module falls behind: it infers on images up to that many inferences old. Each waiting image holds a whole image in memory.

A changed queue size applies to the next image that arrives, with no restart of the stream.

Running a stream

Make sure the Cam Driver’s stream runs: its control/publish/output/stream/active is true.
Subscribe to control/subscribe/output/stream/result.
Write true to control/subscribe/input/enable - and the source’s address, if the module has none yet - and true to control/subscribe/execute. Wait until control/subscribe/output/stream/active is true. The results arrive from then on.
To stop, write false to control/subscribe/input/enable and true to control/subscribe/execute.

Both are saved: the module starts again as it was left. The stream can be started before the Cam Driver’s stream; results arrive once images do.

Inference on request while the stream runs

Both use the same network, so they never run at the same time:

control/inference/ready is false while the stream runs.
A request on control/inference/request is answered with status 100002 (StreamActive). It does not latch an error, and no reset is needed.
control/inference/execute answers DL_INVALID_OPERATION_MODE (LumiScan Keryx: InvalidOperationMode). Do not retry it in a loop: stop the stream first.
Reset, list, load, save, save-result-image and remove keep working. A reset loads the network anew and the stream goes on with it.

Measured rates

Measured with one simulated camera (1440 x 1080, Mono8), a detection network with a 640 x 480 input on an NVIDIA GeForce RTX 2060 SUPER with the CUDA provider, under Linux. The figures show the order of magnitude only.

ctrlX Data Layer: the Data Layer made no difference to the figures.
LumiScan Keryx on a PC (Windows): on the same GPU, an inference took about 53 ms, and 18.4 of 30 images per second were inferred; the table shows the run under Linux.

Camera	Received	Inferred	Skipped	Inference time
10 images/s	10 /s	10 /s	0	about 49 ms
30 images/s	30 /s	19.4 /s	10.7 /s	about 51 ms

Camera

Received

Inferred

Skipped

Inference time

10 images/s

10 /s

about 49 ms

30 images/s

30 /s

19.4 /s

10.7 /s

about 51 ms

No image was lost or failed.

Reading the statistics

The counters are under control/subscribe/output/stream/statistics/. They start from zero each time the stream starts, and are updated about four times a second while it runs.

Counter Meaning

Counter	Meaning
`elapsed-ms`	Milliseconds since the stream started. Divide a counter by it for a rate.
`received-frames`	Images the subscription delivered.
`inferred-frames`	Images inferred, each with a result published on `control/subscribe/output/stream/result`.
`skipped-frames`	Images never inferred: pushed out of the queue by newer ones while inferences ran (see The queue size), or waiting when the source was replaced.
`failed-frames`	Images that could not be decoded or inferred, for example because their pixels do not fill the size and pixel format they carry. The log says why.
`lost-frames`	Images dropped before they arrived, counted from gaps in the camera’s `sequence` within one run of its stream. A new run of the camera’s stream, or a new source, starts the count again.
`inference-us`	Microseconds all inferences took. Divide by `inferred-frames` for the mean.

elapsed-ms

Milliseconds since the stream started. Divide a counter by it for a rate.

received-frames

Images the subscription delivered.

inferred-frames

Images inferred, each with a result published on control/subscribe/output/stream/result.

skipped-frames

Images never inferred: pushed out of the queue by newer ones while inferences ran (see The queue size), or waiting when the source was replaced.

failed-frames

Images that could not be decoded or inferred, for example because their pixels do not fill the size and pixel format they carry. The log says why.

lost-frames

Images dropped before they arrived, counted from gaps in the camera’s sequence within one run of its stream. A new run of the camera’s stream, or a new source, starts the count again.

inference-us

Microseconds all inferences took. Divide by inferred-frames for the mean.