Transforming a Stream (Experimental)

The stream is experimental (from v2.1.0). Its nodes and settings may still change. Use it for evaluation, and verify it with your setup before relying on it.

The transform function converts the points your program writes, one request at a time. A stream works the other way round. The module subscribes to one Data Layer node, its source, and transforms every update of it as it arrives. The source is named in two parts: the folder it is in (address), and the node in that folder (the step’s node). It publishes the result on <module name>/control/subscribe/output/stream/points, with the update’s lineage - where its points came from - and on <module name>/control/subscribe/output/stream/plain-points as a plain array, and your program subscribes to one of them.

A stream can read its points in two ways. The step in the onEntry list of the module’s stream section says which:

  • plain (no step, or a plain step): the node holds the points themselves, as one array of numbers: x0,y0,x1,y1,…​ or x0,y0,z0,x1,y1,z1,…​. A PLC array is a typical source.

  • remap (a remap step): the node holds the result of another application as a flatbuffer - by default the image-stream result of LumiScan AI Object Matching. The step names the list of elements in it, and the fields that hold each element’s x and y.

Either way, the module publishes one triplet per point, x,y,z, in the order of the source (see The points). A plain stream can read another module’s points, so modules chain (see Chaining modules).

A module has one source, transformed with the module’s one lookup table. To transform two sources, use two modules - also for two sources in the same camera’s image, which then share the same lookup table file.

A remap stream transforms every result of its source, whichever camera it came from: a filter reads fields of the elements, not the result’s lineage. To transform the results of several cameras with each camera’s own table, let each camera have an AI Object Matching module of its own, and each of those a Calibration module.

Configuring a stream

Every module has the stream nodes, and every module can stream. The stream section of its module configuration file names the source and holds stream mode. Its onEntry list holds the steps that read each update of the source. A module whose file has no stream section streams plain, and control/subscribe/execute makes the section when it saves the source.

The stream section has the same general keys in every LumiScan application that subscribes to a stream. Only the steps differ:

"stream": {
    "enable": true,
    "address": "hdvisionsystems/lumiscan-ai-object-matching/example/control/subscribe/output/stream",
    "queueSize": 64,
    "keepLatestOutput": false,
    "onEntry": [
        { "kind": "remap", "node": "result", "array": "instances",
          "x": "oriented_bounding_box.center_x", "y": "oriented_bounding_box.center_y" }
    ]
}
Key Required Meaning

enable

No

Stream mode: true runs the stream from the start of the application, without anyone calling control/subscribe/execute. Needs an address. Left out, or false: the stream is off until control/subscribe/execute switches it on.

address

With enable

The folder of the node to subscribe to; the step’s node names the node in it. It is used exactly as written: its case is kept, which a PLC symbol needs. For a plain stream the node must hold a float32 or float64 array, for a remap stream a flatbuffer of the step’s schema. Left out, or empty, to subscribe at run time. An address that ends in the step’s node - the node itself, not its folder - is subscribed to with the node added once more, which fails: the log warns and names the folder to write.

tag

No

The tag on the points the module publishes: the module’s step in their lineage carries it, so a client further on sees which module transformed them. Left out, or empty: the module name. See The lineage of the points.

queueSize

No

How many updates may wait to be transformed: 1 to 4096, 64 by default. See The queue.

keepLatestOutput

No

true also writes the points of each update to the module’s regular output. See Keeping the latest output. false by default.

onEntry

No

The steps applied to each update as it arrives, in order, each an object with a kind: a plain or a remap step. One step is handled for now; a list with more is malformed. Left out, or empty: a plain stream with 2 numbers per point.

onExit

No

Reserved for steps applied to the points before they are published. Not read yet: the module logs a warning, and the steps are not applied.

  • Adding, changing or removing the stream section takes effect with a reset of the module. No restart of the application is needed.

  • A reset that changes how the source is read - a step of onEntry - restarts the stream on the module’s next cycle, about 100 ms later. A reset that changes only enable, address, tag, queueSize or keepLatestOutput restarts nothing: the running stream follows.

  • The section also holds stream mode and the source as set at run time: control/subscribe/execute and control/unsubscribe/execute save them into it. See Kept in the module configuration file.

  • A malformed stream section is a malformed module file: the module reports status -2 (invalidSetting) and transforms nothing until it is corrected and reset. Malformed are, for example, stream mode without a source (enable set to true and no address), a step of another kind, and a key of a step - kind, node, components, schema, rootType, array, x, y or filters - written at the top of the section instead of in onEntry. The log names the key.

An older subscriptions key, which named several sources by tag, is not read any more: the module logs a warning that says so. Move its one source to address - its folder - and the step’s node.

To see what a source may hold, read hdvisionsystems/lumiscan-calibration/system/meta/subscription-schema: see What a source may hold.

Plain: arrays of points

"stream": {
    "enable": true,
    "address": "plc/app/Application/sym/GVL_Vision",
    "onEntry": [
        { "kind": "plain", "node": "aPixelPoints", "components": 3 }
    ]
}

A stream without a step - no onEntry, or an empty one - is plain, with 2 numbers per point, and reads the node points: another Calibration module’s points (see Chaining modules). A plain step is needed for another node, such as a PLC array, or for 3 numbers per point.

Key of the step Required Meaning

kind

Yes

"plain"

node

No

The node of the folder that holds the array, for example a PLC symbol, aPixelPoints. points by default.

components

No

The numbers per point: 2 for x,y (the default), 3 for x,y,z. The lookup table maps a pixel, so a z is ignored. An array whose length is not a whole number of points is not transformed and is counted in statistics/failed-updates.

Remap: points from another application’s result

"stream": {
    "address": "hdvisionsystems/lumiscan-ai-object-matching/<module name>/control/subscribe/output/stream",
    "onEntry": [
        {
            "kind": "remap",
            "node": "result",
            "array": "instances",
            "x": "oriented_bounding_box.center_x",
            "y": "oriented_bounding_box.center_y",
            "filters": [
                { "field": "class_index", "anyOf": [0, 2] },
                { "field": "score", "min": 0.5 }
            ]
        }
    ]
}

This example transforms the centre of each box AI Object Matching finds on its image stream, for the classes 0 and 2 with a score of at least 0.5. It reads the node result of the AI module’s stream folder, a flatbuffer of the default schema, stream_detections.

A stream result such as this one holds two things: its lineage, and the result itself - the same result the application writes to its regular output. The step reads inside the result, so array is the path as on the regular output: instances, AI Object Matching’s list of what it found. The points the module publishes carry the lineage of the AI result - the camera image it was inferred on - with the module’s own step added.

The example module the application ships, example, has such a step without filters. It reads AI Object Matching’s example module, also example, from its start. See Module Configuration File.

Key of the step Required Meaning

kind

Yes

"remap"

node

No

The node of the folder that holds the flatbuffer. result by default: the stream result of a LumiScan application, in its control/subscribe/output/stream folder.

schema

No

Which flatbuffer the node holds; stream_detections, AI Object Matching’s stream result, by default. Either the name of a schema the application ships, or the path of a compiled schema (.bfbs) file, relative to the module configuration file. The schemas shipped, with the array of each:

* the stream results (node result): stream_detections (AI Object Matching, instances), stream_template_matches (Template Finder, matches), stream_object_matches (Object Finder, matches), stream_areas (Area Detector, blobs), stream_codes (the code readers, results), stream_texts (Text Reader, words); * the regular outputs: object_detection (AI Object Matching’s output/result, instances), template_match_result, object_find_result, area_detect_result, code_detect_result and text_read_result, with the same lists; * another Calibration module’s points: stream_points (node points, array points, x and y named x and y).

rootType

No

The table at the root of the buffer, with its namespace, for example hdv.object_detection2d.Result. Leave it out to use the schema’s own root type.

array

No

The path to the list whose elements are the points, for example instances. In a stream result it starts inside the result, as on the regular output. Left out: no list - the result itself is the one element, and each update gives one point, or none when a filter rejects it. See One point per update.

x, y

Yes

The path within one element to its x and its y, for example oriented_bounding_box.center_x.

filters

No

Only the elements every filter passes become points. Each filter has a field (a path within an element to a number) and at least one of anyOf (a list of values the field must equal one of), min and max (inclusive bounds). An integer field is compared as an integer.

A path is a list of field names joined with ., as the schema names them. Each step but the last must lead to a structure, and the last to a number. The module checks every path against the schema when the stream starts. If one does not fit, the stream does not start, and the log says which step is wrong. Two mistakes are named with the fix:

  • an array that starts with the stream result’s own field, result.instances: the result is read already, write "array": "instances";

  • no array while x or y is only found in the elements of a list of the result: name the list, for example "array": "instances".

One point per update

A result that is not a list - one pose, one position - gives one point per update. Leave array out, and name the fields within the result:

{ "kind": "remap", "schema": "my_result.bfbs", "node": "result", "x": "pose.x", "y": "pose.y" }

The filters apply to that one element: an update it does not pass is published with no points.

Good to know:

  • A field the element does not set reads as its default value in the schema, as for any flatbuffer reader.

  • If a structure on a path is missing from an element, the point is published as NaN,NaN,NaN, and a filter on that path rejects the element.

  • An update that is not a valid buffer of the given type is not transformed. It is counted in statistics/failed-updates, and the first one is written to the log.

Addresses

All addresses below start with hdvisionsystems/lumiscan-calibration/<module name>/. Every module has them. The stream follows the layout of an execute method: control/subscribe owns its inputs and outputs.

Table 1. Methods
Address Meaning

control/subscribe/execute

Checks all of control/subscribe/input/…​ together, saves them in the module configuration file, then applies them: the stream follows on the module’s next cycle, with no restart. See Subscribing at run time.

control/unsubscribe/execute

Drops the source named by control/unsubscribe/input/address, which ends stream mode too, and saves that.
Table 2. Inputs
Address Type At start Meaning

control/subscribe/input/enable

bool

stream.enable, or false

true runs the stream; false stops it.

control/subscribe/input/address

string

stream.address, or empty

The folder of the node to subscribe to, exactly as written. The step’s node names the node in it.

control/subscribe/input/tag

string

stream.tag, or empty

The tag on the points the module publishes, in their lineage. Empty: the module name.

control/subscribe/input/queue-size

uint32

stream.queueSize, or 64

How many updates may wait to be transformed, 1 to 4096. See The queue.

control/subscribe/input/keep-latest-output

bool

stream.keepLatestOutput, or false

true also writes each update’s points to the regular output. See Keeping the latest output.

control/unsubscribe/input/address

string

empty

The source control/unsubscribe/execute drops: the address on setting/stream/address.

Writing an input changes nothing by itself: only control/subscribe/execute applies the inputs. They start as what the module applies, and a reset sets them to the file’s values again, so a client can change one input and call the method.

Table 3. Outputs
Address Type Meaning

control/subscribe/output/stream/active

bool

true while the stream runs.

control/subscribe/output/stream/points

flatbuffers (stream_points)

The points of each update, transformed, with their lineage. Subscribe to it to receive each one. A read returns the latest. See The points.

control/subscribe/output/stream/plain-points

float32 array

The same points as a plain array, x0,y0,z0,x1,y1,z1,…​, without the lineage: for a client that cannot read a flatbuffer, such as a PLC.

control/subscribe/output/stream/statistics/…​

uint64

Counters of the current or last stream, see Reading the statistics.
Table 4. Setting
Address Type Meaning

setting/stream/enable, setting/stream/address, setting/stream/tag, setting/stream/queue-size, setting/stream/keep-latest-output

as the inputs

What the module applies now, as saved in the stream section.

setting/settings-hash

string

The hash of what the points are transformed with, as 16 hex digits. See The settings hash.

Unless keep-latest-output is set, the points of the stream are published only on control/subscribe/output/stream/points and control/subscribe/output/stream/plain-points, and the regular outputs keep the result of the last transform on request.

What a source may hold

hdvisionsystems/lumiscan-calibration/system/meta/subscription-schema (string, read only, experimental) describes what the node a module subscribes to may hold, as a JSON Schema (draft-04) of the node’s JSON form. It lists three kinds, any one of which fits (anyOf):

  • a plain array of numbers - a plain step, or none;

  • a stream_points - another Calibration module’s points, read plain;

  • a stream_detections - AI Object Matching’s stream result, read by a remap step with the default schema.

It is the same for every module, so it does not say which kind one module reads: its onEntry step says that. The descriptions in it come from the flatbuffer schemas. A field of a flatbuffer is shown as the JSON of flatc --strict-json renders it: an enum by its name, a structure as an object.

Running a stream

  1. Subscribe to control/subscribe/output/stream/points, or to control/subscribe/output/stream/plain-points for the plain array.

  2. Write true to control/subscribe/input/enable - and the source’s address, if the module has none yet - and true to control/subscribe/execute. Wait until control/subscribe/output/stream/active is true. Updates of the source are transformed from then on.

  3. To stop, write false to control/subscribe/input/enable and true to control/subscribe/execute.

Good to know:

  • Stream mode is saved in the stream section before it applies: after a restart of the application the module streams again, or not, as it did before. Nobody has to call the method again.

  • The stream runs only while the licence is present.

  • If the stream cannot start because a remap path does not fit the schema, control/subscribe/output/stream/active stays false and the reason is written to the log. It is not retried until the module is reset, or stream mode is switched off and on again with control/subscribe/execute.

  • A source that cannot be subscribed to, for example a node that does not exist yet, does not keep the stream from running: it is tried again every 5 seconds, and the first failure is written to the log.

  • Transforming on request keeps working while the stream runs: the transform handshake and control/transform/execute are never refused because of the stream.

  • A reset replaces the lookup table at once, also for the running stream: the next update is transformed with the new table. If the reset fails, the module has no table, and the stream publishes NaN for every point until a reset succeeds.

The queue

The updates of the source wait in a queue, in the order they arrive, and are transformed one after the other. An update is dropped only when the queue is full - queueSize updates, 64 by default, are already waiting: then the oldest goes, and is counted in statistics/skipped-updates.

A larger queueSize rides out longer bursts without dropping an update, but when the source updates faster than the module transforms for longer, the points it publishes are older: up to queueSize updates behind. A smaller one keeps the points recent and drops more.

A changed queue size applies at once, with no restart of the stream. When it shrinks below the updates already waiting, the oldest of them are dropped and counted as skipped.

Subscribing at run time

A PLC or another client changes the source and stream mode while the application runs, with or without a running stream:

  1. Write the inputs that change to control/subscribe/input/…​ - for another source, control/subscribe/input/address, its folder (the node in it stays the step’s); to start or stop, control/subscribe/input/enable; for another tag on the points, control/subscribe/input/tag.

  2. Write true to control/subscribe/execute.

The method checks all inputs together, saves them in the stream section, then applies them: setting/stream/…​ shows them at once, and a running stream follows on the module’s next cycle. Another address replaces the source - no unsubscribe is needed first -, and the updates the old source had waiting are dropped and counted as skipped. The queue size and keep-latest-output apply without subscribing again, and so does the tag: the points published from then on carry it.

To drop the source, write its address to control/unsubscribe/input/address and true to control/unsubscribe/execute. That ends stream mode too, and saves enable as false with no address. The tag stays: it marks the module’s points, not the source. control/subscribe/input/…​ are set to that as well, so a later control/subscribe/execute that changes another input does not bring the source back. A / at either end of an address does not matter.

The methods answer as follows. Where a result has one name only, it is the name on the ctrlX Data Layer; LumiScan Keryx reports the same result under its own name, see Results of a method call.

Result When

DL_OK

Done.

DL_INVALID_VALUE (ctrlX Data Layer), Rejected (LumiScan Keryx)

control/subscribe: enable is true and the address is empty, or the queue size is not 1 to 4096. control/unsubscribe: the address is not the source, or there is no source. Nothing changed.

DL_CREATION_FAILED

The change could not be saved in the module configuration file: the file could not be read or written. Nothing changed: the stream stays as it was. The answer and the log say why.

DL_RESOURCE_UNAVAILABLE

The settings were edited on system/settings and the application waits for a restart.

Kept in the module configuration file

Stream mode and the source are settings of the module, in the stream section of its configuration file. control/subscribe/execute and control/unsubscribe/execute save enable, address, tag, queueSize and keepLatestOutput there before they apply them: when they cannot be saved, the method answers DL_CREATION_FAILED and nothing changes. After a restart the module subscribes to the same source and, when enable is true, starts its stream at once.

Saving changes only those values: the rest of the file stays as it was, comments and the steps of onEntry included. A file without a stream section gets one, with those values and no step: a plain stream.

A reset reads the file again, and sets the inputs to it: a change you make in the file by hand takes effect with the next reset.

Keeping the latest output

With keep-latest-output set, the points of each update are also written to the module’s regular output, as a transform on request writes them: output/x-coordinate-array, output/y-coordinate-array and output/z-coordinate-array, output/timestamp, and output/package-identifier, counted on from the transforms on request. A client that reads the regular output - a PLC program written for the transform function, for example - then sees the newest points of the stream there. A point that cannot be transformed is NaN there too.

It costs a copy of every update’s points, so it is off by default. The points on control/subscribe/output/stream/points are published first.

The points

control/subscribe/output/stream/points holds a FlatBuffers table of type stream_points, registered on the Data Layer as types/hdvisionsystems/lumiscan-calibration/stream_points. Its compiled schema is installed as bfbs/stream_points.bfbs.

Field Meaning

points

One x, y, z (float32) per point of the update, in its order.

invalid

How many of the points could not be transformed.

lineage

Where the update came from, see The lineage of the points.

control/subscribe/output/stream/plain-points publishes the same points as x0,y0,z0,x1,y1,z1,…​, right after control/subscribe/output/stream/points.

  • The i-th point is the i-th point of the update. A point that cannot be transformed is published as NaN in its place, so the order is kept. A point cannot be transformed when it lies outside the lookup table, or when the module has no table.

  • An update with no points is published with none.

  • z is the lookup table’s value. A z in a plain source with components 3 is ignored.

A point outside the lookup table is published as NaN by the stream only. On request, the transform function returns a value from the edge of the table for such a point, without an error, as it always did. Keep the points you request within the image the table was made for.

Chaining modules

A module’s control/subscribe/output/stream/points can be the source of a plain stream in another module. The second module reads it as x,y,z points, whatever its components say, transforms the x and y of each, and ignores the z. Its points carry the first module’s lineage with its own step added. control/subscribe/output/stream/plain-points works as a source too, with components 3, but its points start a lineage of their own.

The lineage of the points

lineage.steps lists where an update’s points came from, origin first:

  • A source that carries a lineage - AI Object Matching’s control/subscribe/output/stream/result, another module’s control/subscribe/output/stream/points - passes its steps on: the camera’s step first (producer the camera’s address, tag the Cam Driver module’s tag, run and sequence the image’s place in the camera’s stream), then the AI module’s, with the AI module’s tag.

  • A source that carries none - a PLC array - gets a step made for it, marked synthesized: its producer is the node’s address, the folder and the node; it has no tag; its sequence the module’s own count since the stream started. On LumiScan Keryx, the sequence is the Data Layer’s sequence number of the update where there is one, else the module’s own count.

  • The module’s own step comes last: producer is the module’s address, hdvisionsystems/lumiscan-calibration/<module name>; tag the module’s tag (stream.tag), or the module name when it has none; run when the module’s stream started; sequence the update’s number in the run; settings_hash, see The settings hash; elapsed_us the microseconds from the update’s arrival to the points' publication.

The tag of each step says which module processed the update there - the module’s own name, or the tag it was given. Give two modules of the same name in different applications, or a module whose name says little, a tag that says what it does on your line.

A source changed while the stream runs keeps the module’s run: the updates of the new source are numbered on, and the producer of their first step names it.

The settings hash

setting/settings-hash, and the settings_hash of the module’s step, identify what the points are transformed with. The hash changes when one of these changes:

  • the module configuration file’s settings - except the transform’s name; the stream’s enable, address, the node of its step, tag, queueSize and keepLatestOutput, which switch, choose, pace and mark the updates rather than change their points; and its onExit, which is not read yet;

  • the lookup table (relativePathToTiff) by its content - a new table under the old file name is a new hash; and the schema of a remap step, by its content - the same schema under another name is the same hash;

  • the application’s version.

A reset reads them again. The first time a hash appears, the module writes the settings it stands for to settings-history/<module name>-<hash>.json in the application’s configuration folder, so a hash found in the points' lineage can be looked up later.

The points carry the hash only. A different hash says that something they are transformed with changed, not what changed. To find out, compare the two files in settings-history that the hashes name, by hand.

Reading the statistics

The counters start from zero each time the stream starts, and are updated about four times a second while it runs. All addresses start with control/subscribe/output/stream/statistics/.

Counter Meaning

elapsed-ms

Milliseconds since the stream started. Divide the other counters by it for rates.

received-updates

Updates of the source that arrived.

transformed-updates

Updates transformed and published on control/subscribe/output/stream/points and control/subscribe/output/stream/plain-points.

skipped-updates

Updates dropped unprocessed because the queue was full (queueSize, 64 by default): the module could not keep up with the source. The oldest waiting update is dropped first. Also the updates waiting when the source was replaced, or when the queue size shrank below them.

failed-updates

Updates that could not be transformed: not the expected type or length, or not a valid buffer. The first one is written to the log.

lost-updates

Updates dropped before they arrived. For a source that carries a lineage, counted from gaps in the sequence of its producer’s step within one run. On LumiScan Keryx, for a plain array, from gaps in the Data Layer’s sequence numbers. A new source starts the count again.

points

Points transformed, all updates together.

invalid-points

Of those, the points published as NaN.

transform-us

Microseconds all transforms took, summed. Divide by transformed-updates for the mean.

What a stream receives

control/subscribe/output/stream/points publishes every update to its subscribers. What the stream itself receives depends on the source node:

Data Layer Source that publishes each update (for example AI Object Matching’s control/subscribe/output/stream/result, or another module’s control/subscribe/output/stream/points) Ordinary node (for example a PLC array)

ctrlX Data Layer (ctrlX OS)

Every update, in order.

The value about every 100 ms. Changes in between are not seen.

LumiScan Keryx (PC)

Every update, in order.

Every change: writing the same values again is not an update.

On the ctrlX Data Layer, the stream has not yet been measured with a PLC source on a device.

The stream concept of all LumiScan applications is described in Streams (Experimental).