Streaming Images (Experimental)

The image stream is experimental (from v2.3.0). Its nodes may still change. Use it for evaluation, and verify it with your camera before relying on it.

capture delivers one image per request, and each request waits until its image has arrived. On the camera we tested this gives about 34 images per second, well below what the camera itself can deliver: the next exposure cannot start before the previous image has been read out and sent.

The image stream works the other way round. While it runs, the Cam Driver publishes every image the camera delivers on <camera name>/control/publish/output/stream/frame. Your program subscribes to that node and is sent each image as it arrives - with its size, its pixel format and where it came from, see What each frame carries.

The stream is switched on and off by one function, control/publish: set its inputs, then call control/publish/execute to apply them. See Running a stream.

The general stream concept - producers, consumers and the lineage of their results - is described in Streams. This page describes the Cam Driver’s stream.

The Cam Driver does not trigger the camera by itself. An image comes when the camera is triggered - by a signal on its trigger input, or by control/trigger/execute - or, with the camera’s trigger mode off, as fast as the camera runs. See Triggering the camera.

Table 1. What a subscriber receives
Data Layer Images delivered to a subscriber of control/publish/output/stream/frame

ctrlX Data Layer (ctrlX OS)

Every image, in order, as soon as it is published, whatever the subscription’s publish interval. Images are dropped only when the Data Layer falls far behind.

LumiScan Keryx (PC)

Every image, in order. The Data Layer drops images only when a subscriber falls far behind.

Each frame carries its number in the stream, so a subscriber sees a dropped image as a gap. A read of control/publish/output/stream/frame always returns the latest image.

Addresses

All addresses below start with hdvisionsystems/lumiscan-cam-driver/<camera name>/. They follow the rule of every execute method: the method owns its nodes, control/publish/execute, control/publish/input/…​ and control/publish/output/…​.

Table 2. The method and its inputs
Address Type Default Meaning

control/publish/execute

method

Applies the inputs below together. Writing an input alone changes nothing. Returns at once: the stream starts or stops within the Cam Driver’s next cycle, and control/publish/output/stream/active says when it runs. Refused with DL_RESOURCE_UNAVAILABLE while the Cam Driver shuts down, or after its settings were changed on the Data Layer (system/restart-required). Refused with DL_CREATION_FAILED when a changed tag cannot be saved: nothing changes then.

control/publish/input/enable

bool

false

true runs the stream while the camera is connected; false stops it.

control/publish/input/keep-latest-output

bool

false

true copies every streamed image to output/image as well: data, width, height, pixel-type and timestamp (the time the image arrived), and then increases output/package-identifier by one, as capture does. Each copy takes time, which slows a fast stream down. Applied to a running stream without restarting it. See Capture while the stream runs.

control/publish/input/tag

string

stream.tag of the module configuration file, or empty

The tag on the module’s images: the camera’s step in their lineage carries it, and so do the results of every application that processes them. Empty: the camera name, for example camera1. Give it another, for example the station the camera looks at, to see that in every result. Applied to a running stream without restarting it. See The tag.

What execute applied is not saved, except the tag: after the Cam Driver starts, the stream is off until control/publish/execute is called with enable true.

Table 3. The method’s outputs, under control/publish/
Address Type Meaning

output/stream/active

bool

true while the stream runs.

output/stream/frame

flatbuffers (stream_image)

Each streamed image: the pixels, their size and format, and the image’s lineage. Subscribe to it to receive the images. See What each frame carries.

output/stream/width, output/stream/height

uint64

Size of the streamed images in pixels, for browsing. Each frame carries its own.

output/stream/pixel-type

string

Pixel format of the streamed images, for browsing. Each frame carries its own.

output/stream/statistics/…​

uint64

Counters of the current or last stream, see Reading the statistics.
Table 4. Beside the method
Address Type Meaning

setting/settings-hash

string

The hash of the settings the images are taken with, as 16 hex digits. See The settings hash.

setting/stream/tag

string

The tag control/publish/execute applied last, as the images carry it. Empty: the images carry the camera name.

The tag

A changed control/publish/input/tag is saved in the module configuration file, as stream.tag, before control/publish/execute applies it, the rest of the file as it was. So the images carry it after a restart too, and control/publish/input/tag starts with it. If the file cannot be written, control/publish/execute answers DL_CREATION_FAILED, and neither the tag nor the other inputs change. An unchanged tag is not saved, so a module whose file cannot be written still streams.

The tag is not part of the settings hash: it names the images, it does not change them.

Triggering the camera

Each trigger the camera receives gives one image on control/publish/output/stream/frame. Set the camera’s trigger mode before you start the stream, through its feature nodes under features or in a feature file.

Images come Camera settings What to do

On a hardware trigger

TriggerMode On, TriggerSource a trigger input, for example Line0

Send a pulse to the trigger input for each image.

On request

TriggerMode On, TriggerSource Software

Call control/trigger/execute once for each image. The Cam Driver sends the camera’s software trigger, as hdv’s feature mapping names it.

Continuously

TriggerMode Off

Nothing. The camera delivers as fast as it runs, or at its AcquisitionFrameRate where that is enabled.

A camera without a software trigger mode - a simulated camera playing a folder of images, or a plugin camera without TriggerSoftware - takes one image for each call to control/trigger/execute as well.

control/trigger/execute answers when the trigger has been sent, not when its image has arrived. A camera that is still busy with the previous image can ignore a trigger without reporting it, see For the highest frame rate: TriggerOverlap.

For the highest frame rate: TriggerOverlap

With the GenICam feature TriggerOverlap set to Off, a camera takes the next image only once the previous one has been read out, and it ignores a trigger that comes earlier. An overlap lets the next exposure start while the previous image is still read out or sent, which raises the frame rate the camera reaches.

To stream at the highest frame rate, TriggerOverlap must not be Off. Set it to an overlap your camera supports - PreviousFrame or ReadOut, see its documentation - in a feature file of the camera module, one feature per line:

TriggerOverlap PreviousFrame

List the file in the module’s cameras[].featureFiles. The Cam Driver applies it each time it connects the camera, before acquisition starts, so it holds for capture as well. A feature node under features works too, but some cameras refuse trigger features while they acquire, and the value is lost when the camera is connected again.

What each frame carries

control/publish/output/stream/frame holds a FlatBuffers table of type stream_image, registered on the Data Layer as types/hdvisionsystems/lumiscan-cam-driver/stream_image. Its compiled schema is installed as bfbs/stream_image.bfbs.

Field Meaning

image.data

The pixels, row after row, without padding.

image.width, image.height

The image size in pixels.

image.pixel_type

The pixel format, for example Mono8.

lineage.steps

Where the image came from. For a camera image this is one step, the camera’s own:

* producer: the camera’s address, hdvisionsystems/lumiscan-cam-driver/<camera name>. * tag: the module’s tag, setting/stream/tag, or the camera name when it is empty. See The tag. * run: when the stream started, in microseconds since 1970. Each start of the stream is a new run. * sequence: the image’s number in the run, from 1. A gap means images were lost on the way. * external_id: the camera’s own frame number (the GigE Vision block id). A gap here without a gap in sequence means images were lost between the camera and the Cam Driver. * time_us: when the image was published, in microseconds since 1970. * settings_hash: the settings the image was taken with, see The settings hash.

The applications that process the stream - AI Object Matching, Calibration - copy the lineage of what they receive and add a step of their own, with their own tag. Their results therefore still name the camera image they came from, by producer, run and sequence, and each step says by its tag which module made it.

The settings hash

setting/settings-hash and the settings_hash of each frame identify the settings the camera works with. The hash changes when one of them changes:

  • the module settings and the feature files, as the camera was connected with;

  • every camera feature written while the camera is connected, for example ExposureTime through input/exposure-time or a feature node under features, with the value the camera accepted;

  • the Cam Driver’s version.

Settings that only affect how images are transported, such as the timeout or the packet size, do not change the hash. The first time a hash appears, the Cam Driver writes the settings it stands for to settings-history/<camera name>-<hash>.json in its configuration folder, so a hash found in a result can be looked up later.

A camera takes the images that are already on their way with the old settings. The images that arrive right after a feature was written can carry the new hash although they were taken before the change.

What the hash does not tell

  • That, not what. A frame carries the hash only. A different hash says that something the images are taken with changed, not what changed. To find out, compare the two files in settings-history that the hashes name, by hand. The files are on the device the Cam Driver ran on.

  • Only what goes through the Cam Driver. The hash follows a feature node under features, input/exposure-time, and the settings and feature files the camera was connected with. It does not follow values the camera changes by itself: with ExposureAuto on, the hash follows the ExposureAuto setting, not the exposure times the camera picks.

That the hash follows a camera feature change has so far been tested only in unit tests, with a stand-in for the camera, not with a real camera. Until it has been, do not rely on the camera’s settings hash alone to show that a camera feature changed.

Running a stream

  1. Connect the camera with the connect-camera handshake. See Connection Steps.

  2. Set the camera features you need, such as the exposure time, the trigger mode and TriggerOverlap, before starting the stream. See Triggering the camera.

  3. Subscribe to control/publish/output/stream/frame.

  4. Set control/publish/input/enable to true, and control/publish/input/keep-latest-output to true if you also want each image on output/image.

  5. Call control/publish/execute, and wait until control/publish/output/stream/active is true.

  6. Trigger the camera. Each image arrives on control/publish/output/stream/frame.

  7. To stop, set control/publish/input/enable to false and call control/publish/execute again. control/publish/output/stream/active turns false within the next cycle.

  8. Before capturing again, wait until control/capture/ready is true. See After the stream.

Good to know:

  • The inputs take effect only on execute, all together. A program can therefore set them in any order, and nothing changes half-way.

  • What execute applied stays until the next execute: the stream runs for as long as enable was applied true and the camera is connected. Applied while the camera is disconnected, the stream starts as soon as the camera is connected.

  • connect-camera, disconnect-camera and reset stop a running stream. If enable was applied true, the stream starts again once the camera is connected.

  • If the stream cannot start, control/publish/output/stream/active stays false and the reason is written to the log. It is not retried until control/publish/execute is called again.

  • A camera ignores triggers that come faster than it can take images, without reporting it. If statistics/frames grows more slowly than the number of triggers you sent, trigger less often or set an overlap, see For the highest frame rate: TriggerOverlap.

  • The exposure time on input/exposure-time and the inter-packet delay are not applied while the stream runs. Change camera features only while the stream is stopped.

Capture while the stream runs

The stream and capture use the camera in different ways, so they never run at the same time:

  • control/capture/ready is false while the stream runs, so a program using the handshake waits.

  • control/capture/execute answers DL_INVALID_OPERATION_MODE while the stream runs, and DL_TOO_MANY_OPERATIONS while the stream is starting or stopping. Do not retry DL_INVALID_OPERATION_MODE in a loop. Stop the stream and wait for capture/ready instead. See Execute a Method.

  • A capture request that reaches the Cam Driver while the stream runs is refused with status -3 (invalidInput), with the text "capture refused: the image stream (experimental) is active". It does not latch a fault, so no reset is needed.

  • A stream that is switched on while a capture runs starts once that capture has finished. It never cuts a capture short.

Streamed images are published on control/publish/output/stream/frame. They reach output/image only while keep-latest-output is applied true: then each one replaces the image there, and output/package-identifier counts it as it counts a capture, continuing from the last capture’s number. save keeps saving the last image taken with capture or load, also then.

After the stream

While streaming, the camera delivers its images in a different acquisition mode from the one capture uses. To switch back, the Cam Driver disconnects the camera and connects it again when the stream stops, applying every connection setting as connect-camera does. capture/ready turns true once that has finished. Until then, capture/execute answers DL_TOO_MANY_OPERATIONS.

Starting and stopping therefore take different times. On the camera we tested:

Step Took Mostly spent on

Starting the stream

about 0.35 s

Switching the camera’s acquisition mode.

Stopping the stream, until capture/ready is true again

about 2.2 s

Connecting the camera again: opening the device (about 0.8 s) and reading all of its GenICam features for the Data Layer (about 0.8 s).

So switch between streaming and capturing as rarely as you can. A program that needs single images while streaming can take them from control/publish/output/stream/frame instead of stopping the stream, or apply keep-latest-output and read output/image when output/package-identifier changes.

Measured frame rates

These figures were measured with a LUCID PHX016S (1440 x 1080, Mono8, exposure 1 ms) and LumiScan Keryx on one PC. They show the order of magnitude only: your camera, exposure time, network and Data Layer decide what you reach.

They were measured while the Cam Driver still triggered the camera itself, at a fixed rate. A hardware trigger or a continuously running camera is expected to give the same. Triggering with control/trigger/execute adds one method call per image; that has not been measured yet.

Method Images per second delivered

capture/execute in a loop

about 34

Stream, TriggerOverlap Off

up to 63, every image delivered

Stream, TriggerOverlap PreviousFrame

up to 66, every image delivered

What the camera itself can deliver (AcquisitionFrameRate)

71.7

Reading the statistics

The counters, under control/publish/output/stream/statistics/, start from zero each time the stream starts, and are updated about four times a second while it runs.

Counter Meaning

elapsed-ms

Milliseconds since the stream started, measured by the Cam Driver. Divide a counter by it for a rate.

frames

Complete images the camera delivered.

incomplete-frames

Images the camera delivered incomplete. They are not published.

lost-frames

Images that never reached the Cam Driver, counted from gaps in the camera’s frame numbers.

published-frames

Images written to control/publish/output/stream/frame.